Cross-entropy

Course Q&A Machine Learning Specialization Supervised ML: Regression and Classification
1

https://www.coursera.org/learn/machine-learning/lecture/0hpr8/cost-function-for-logistic-regression.

In the above video, the loss function is mentioned as below -
𝑙𝑜𝑠𝑠(𝑓𝐰,𝑏(𝐱(𝑖)),𝑦(𝑖))=
−log(𝑓𝐰,𝑏(𝐱(𝑖))) if 𝑦(𝑖)=1
−log(1−𝑓𝐰,𝑏(𝐱(𝑖))) if 𝑦(𝑖)=0

It looks like we only use the predicted value[𝑓𝐰,𝑏(𝐱(𝑖))] to calculate the cost when we are using the above cross-entropy.

Now the question will the loss function return the same loss as long as the 𝑓𝐰,𝑏(𝐱(𝑖)) is same irrespective of what we are predicting.

For an example say we have 2 separate problems -

  1. We are predicting whether the tumor is malignant and y is 1(malignant) and 𝑓𝐰,𝑏(𝐱(𝑖)) is 0.3 for a data point.
  2. We are predicting whether or not spam and y is 1(spam) and 𝑓𝐰,𝑏(𝐱(𝑖)) is 0.3 for a data point.

Do we get the same loss when we use the above-mentioned cross-entropy loss function for both the predictions in the above example?

2

Hi @tamalmallick, the two examples you gave are from two different supervised learning tasks - therefore, their weights are expected to be different. However, if for the i-th example they give the same predicted probability of 0.3 and have the same ground truth of y=1, the loss function on the i-th example will be exactly the same (assuming both are supervised binary classification with cross-entropy loss).

Independently, the total loss over the entire data set is the sum of the losses over individual examples. For the two supervised classification problems the losses over the rest of the training set may not be identical even if the losses are equal for the i-th example because: 1/ the predictions may be different for the other points or 2/ the ground truths y may be different for the other points.

3

Thank you so much for such a wonderful explanation!