Picking the threshold point in the recall/precision curve

Course Q&A Machine Learning Specialization Advanced Learning Algorithms
4

Hi @Amira_Al-Samawi,

They are not contradictory.

Cross validation compares metric performance over models of different sets of hyperparameters. However, we should exclude “threshold” from such set of hyperparameters when the metric in question is precision or recall, because for example tuning the threshold down always increase recall.

You always achieve 100% recall when you set threshold equal to 0 even if your any other hyperparameters are completely non-sense. In other words, since reducing threshold always increase recall, nobody should care to tune other hyperparameters with the technique of cross-validation to achieve the goal of best recall.

We can divide metrics into 3 categories when threshold is concerned:

  1. metrics that are monotonically increasing / decreasing with threshold, e.g. precision / recall
  2. metrics that are independent of threshold, e.g. AUC (google “AUC Area under curve metric” for more)
  3. other metrics

For type 1, we exclude threshold from cross validation. For type 2, tuning the threshold has no effect at all. For type 3, we can include threshold in cross validation.

Raymond

1 Like