In week 3’s lectures, HLP and the problem of label consistency are explained in detail and some examples are provided where it’s suggested that the labelers conform to a common set of rules when providing labels. For example, in the case of identifying phone defects, a rule that could resolve ambiguity would be to label any scratch larger than 0.3mm as a defect. This kind of method is expected to raise HLP and improve the ML system performance as well. But doesn’t this method essentially impute a/the learning rule we would otherwise want the learning algorithm to find into the dataset definition? In other words, when creating deterministic rules of that form, isn’t it natural for the learning algorithm to more easily catch up to (and probably just verify/reinforce) these rules?
Hi Giannis_Antoniadis,
It is precisely the problem that the ML system does not learn such a rule well when the labels that are fed during training are inconsistent. So, for example, the ML system may not be able to correctly predict scratches of 0.35 mm as indicating a defect if some labelers do not label it as such. The incorrect labelling distorts the calibration of the ML system.