Why "logit" stands for the output of linear activation function

Course Q&A Machine Learning Specialization Advanced Learning Algorithms
1

When exploring the optional lab “Softmax function”, I found a statement like this: In the preferred organization the final layer has a linear activation, and for historical reasons, the outputs in this form are referred to as logits .

Then I googled the word “logit” to understand it further, and the explanation is that logit model is equivalent to logistic model.

So I’m a little confused. Why the “logit” stands for the output of a linear activation function rather than a sigmoid activation function?

2

The “from logits” method takes a linear output and applies logistic softmax detection automatically

1 Like
3

If you would like to read some maths, you might check out this discussion.

4

Thanks, @TMosh and @rmwkwok .

The statement “logistic softmax automatic detection” and the math about how to deal with excessive exponential inspired me a lot.

But I’m still a little confused about the origin of the name of logit. I just don’t understand the naming logic. Why don’t we just use “from_linear” rather than “from_logits” to represent that the input to the loss function is just a linear activation? Or is there something I don’t understand clearly?

5

Hello @Yulin_Li,

I think logits can be defined as the inverse of sigmoid:

logit(p) = \log{(\frac{p}{1-p})}
\implies \exp{(logit(p))} =\frac{p}{1-p}
\implies p = \frac{1}{1+\exp{(-logit(p))}}
\implies p = \frac{1}{1+\exp{(-z)}}
\implies p = sigmoid(z)

where z = logit(p) = linear(a) and linear refers to the linear activation. Therefore, z is called the logit.

Raymond

3 Likes
6

Hello @rmwkwok ,

Great thanks to you!

These formulas solve my doubts very well. And I’ll set this as the solution.

7

You are welcome Yulin!

9

So, from_logits=True will extract the value of p,

which is sigmoid(z), right?

10

Setting from_logits=True for a tensorflow loss function means that the function expects for z. It does not extract. Please google for how people use it, and practice it yourself.

image

p is sigmoid(z). @farhana_hossain , please be explicit about the subject of your statement too, next time.

Cheers,
Raymond

11

Yes, I got it at first.

I will make sure this thing