What is the name of entropy function?

Course Q&A Machine Learning Specialization Advanced Learning Algorithms
1

The entropy function looks like the sigmoid loss function to me

image
image891×405 47 KB

The name of sigmoid is cross entropy, what is the name entropy function (for me it is another loss function)

2

Also one more question, here probability formula is of naive probability, right?

P(X) = \frac{\# \ favourable \ outcome }{\# \ total \ items} \\ \ \\ P(\bar X) = 1- P(X)
3

Hi @tbhaxor

what do you mean specifically with sigmoid loss function? (The entropy profile you shared has nothing directly to do with the sigmoid activation or the context of logistic regression if this is the direction of your question | apart that the output of the sigmoid function in a logistic regression could be interpreted as probability estimate which then could serve as input for the Shannon entropy]. On an abstract level one can say: both, the sigmoid function and the Shannon entropy can be used to describe the uncertainty of a prediction resp. a probability.

Frankly, I am not aware of an official recognised name of this function for this entropy profile you were showing. Still I believe it’s important to understand since the Shannon entropy plays a crucial role in many applications, especially in AI.

Think of of random experiment - a coin toss:
(Here your definition of the 2nd post also suits!)

  • the curve says that if you have a fair coin (50/50 chance) you have a maximum entropy as a rather high level of uncertainty or randomness in this case
  • if the coin is manipulated (in either direction) the entropy (and the randomness) will decrease
  • if the coin is manipulated extremely so that the result is deterministic, the entropy is 0 (as is the randomness)

Hope that helps!

Best regards
Christian

4

For example here: the binary cross entropy loss function would correspond to your plot, since in the end it’s the same formula underlying (Shannon entropy definition):

https://www.researchgate.net/figure/Graph-of-Binary-Cross-Entropy-Loss-Function-Here-Entropy-is-defined-on-Y-axis-and_fig2_342520628

Best regards
Christian

5

It not linked with sigmoid but it looks like that

6

Feel free to take a look at the sigmoid function:

  • Sigmoid function - Wikipedia
  • Besides the different shape, also the input range is all real numbers and not limited to the interval of [0…1].

Best regards
Christian