Week 3: Improving HLP for the tasks where it is hard to create labeling instructions

Course Q&A Machine Learning Engineering for Production(MLOps) Introduction to Machine Learning in Production
1

How can we improve HLP for labeling unstructured text data for which it is hard to enumerate all the labeling instructions?

For example, we have a short answer grading dataset, where one grader can assign 4/5 and the second being lenient to give 5/5. In this case there can be so many factors (Example: grammar, topics answered, terms answered and so on) that can affect a grade, particularly when multiple graders assigning the grade. Also, it is necessary to attain labelers who have similar level of understanding and skills. Even if there are labeling instructions, it doesn’t make sense to stick to these cases as there can be multiple ways to provide a correct answer. In the cases like these, what are the best approaches to improve HLP?

2

Hi @gsasikiran , welcome to the course!

In my opinion, if we cannot have good instructions or the labelers cannot follow the instructions then there is not much we can do here to improve the HLP.
A more practical approach is to receive feedback from production deployment and then do the analysis (tagging which causes bad performance like in the course) then improve the data labeling iteratively.

There may be a better solution, but that’s all I can offer.
Cuong

1 Like