Hi @tbhaxor
what do you mean specifically with sigmoid loss function? (The entropy profile you shared has nothing directly to do with the sigmoid activation or the context of logistic regression if this is the direction of your question | apart that the output of the sigmoid function in a logistic regression could be interpreted as probability estimate which then could serve as input for the Shannon entropy]. On an abstract level one can say: both, the sigmoid function and the Shannon entropy can be used to describe the uncertainty of a prediction resp. a probability.
Frankly, I am not aware of an official recognised name of this function for this entropy profile you were showing. Still I believe it’s important to understand since the Shannon entropy plays a crucial role in many applications, especially in AI.
Think of of random experiment - a coin toss:
(Here your definition of the 2nd post also suits!)
- the curve says that if you have a fair coin (50/50 chance) you have a maximum entropy as a rather high level of uncertainty or randomness in this case
- if the coin is manipulated (in either direction) the entropy (and the randomness) will decrease
- if the coin is manipulated extremely so that the result is deterministic, the entropy is 0 (as is the randomness)
Hope that helps!
Best regards
Christian