in the introduction video, Andrew Ng said if the model do well in the test set, but perform not well in the real app, then we need to change the cost function or change the dev set distribution.
l am confused in why we need to change the dev set not the test set? because l think the model is overfiting the test set but not generalize to the real work data.
Hey @jiahengchen,
I guess that is based on the assumption that the test set is correct and close to the true data distribution. We don’t strictly require that the dev set also come from the same distribution as the test set, but to achieve a better generalization these distributions need to be close. I believe this idea Andrew was talking about in the video.
Hello Andrei,
In the case where we have found out it is necessary to “change the dev set”, then we would be dealing with 3 different data distributions / sources (test, dev, train), right?
Thanks a lot!
On the opposite, we want our dev and test sets to be close to the real-data distribution, so we’re fixing the dev set.