I was under the impression that error analysis should be done on the training or train-dev set, given that the Bayes error is originally defined for the training set distribution.
But after taking the week quiz it signals dev as the correct answer. I re-watched the video on error analysis but I’m unable to catch if the prof. is talking about the dev or train-dev set.
This has me also thinking what happens when we do error analysis on a dev set that suffers from data mismatch. We might only find out the data has a mismatch problem after we have spend resources reducing bias.
I think this problem might be mitigated calculating human level performance on a dev-dev set (?) and GenAI seems to suggest this. However that topic has not been discussed on the course and I would like to know if dev set hlf is a common pattern to tackle missmatch detection or if there are other ways to apply the tools of the course.
Hi! In terms of evaluating the model’s performance, error analysis is usually done on the dev set, because the model has already seen the training data during training. Therefore, evaluating on the training set may not give you a realistic picture of how well the model generalizes to unseen data.
Regarding data mismatch, ideally this should be addressed before training starts. When splitting your data into training, dev, and test sets, you should make sure that they come from the same or very similar distributions and represent similar types of data. Randomization during the splitting process can also help ensure that the sets are representative.
So, ideally, the data distribution issue should be identified and handled before you begin training and error analysis. Otherwise, you may end up spending resources trying to reduce bias or other errors when the underlying issue is actually a mismatch between the datasets.
If I understand correctly, the problem to mitigate here is inspecting a dev set without knowing it has data mismatch. Let’s pivot from what the lectures are telling us:
First, the lecture below shows that we can identify a data mismatch between “dev” and “train / train-dev” by just comparing the errors. With this, before we start inspection, we already know that there is/isn’t a data mismatch.
Second, inspecting such dataset is helping, of course with in mind the existence of data mismatch, because the lecture below also suggests to carry out analysis to understand the difference:
I think @gent.spah 's advice to “make all those sets similar” captures the idea that the second bullet point delivers. One difference is that if you were working on a production system, you don’t start from the beginning.
