As the lesson from course 2 that our dev test set should come from same distribution. But when we apply it in real world, we can not control the distribution of data. For example, in image classification, our model should be able to classify cat and not cat from any picture taken from any camera, and light condition,… This mean we can not control the distribution of real world data, so can our model get best result in real world picture?
Hey @khoi2406,
Welcome to the community. The answer to this question relies on how well your dev set represents the test set. As you said, in real world, it is not possible to control the distribution of the data. But if we try to include images of as many cat breeds as possible, try to click images from different sensors with different settings, try to include images shot under different lighting conditions in our dev set, and even try to include adversarial examples, I don’t see why the model may not get the “best” performance in real-world. Note that here, I am assuming “best” refers to better than human-level performance. I hope this helps.
Regards,
Elemento
1 Like
Thanks for your explanation, @Elemento
So that my model just can work well with picture that has same distribution with my training data. With other picture condition, my model will not better than random guess.
What if I use test set with different distribution with dev set to evaluate how my model work with unseen data? Is it good way to evaluate model?
Hey @khoi2406,
Even in the case of different image capturing conditions, your model will certainly work better than random guess, since the model would have at least learned something useful for making predictions even in the worst conditions. For instance, the whiskers of cats could be a distinguishing feature for cats, which could be captured even in worse conditions.
As to your second query, using a test set having a different distribution than the dev set to evaluate the model, is like training the model to hit target A, and then once it is trained, asking it to hit target B. As far as I recall, it is the same analogy that Andrew used in the lectures. Moreover, the test data is in fact unseen data only, since the model neither has been trained on test set, nor has been optimized using the test set. I guess what you really want to ask is how to find out the performance of the model on a dataset having an unseen distribution.
And this is the line where we are stepping into the real world. In the real world, we have something known as production data, which we get only after the model is deployed. For this dataset, we don’t know what the distribution will be, although, we try our best that to create dev/test sets such that it’s distribution is as close to the test/dev sets as possible. So, naturally, you will get to see the performance of the model when the model is deployed, and once you have acquired enough production data, just revise your dev/test sets to reflect the distribution of the production set, and this cycle keeps on going. I hope this helps.
Regards,
Elemento
Now I know why dev and test set have to come frome same distribution
Thank you so much for your clear explanation.
1 Like