Closeness of devset to trainset is a bad indicator?


When teacher wants to say augmentation does not always help, mentioned : “What’s happening here is that despite the image augmentation, the diversity of
images is still too sparse and the validation set may also be poorly designed, namely that the type
of image in it is too close to the images
in the training set.”

I always thought that different splits (train,dev,test) of data should have same distribution and this closeness is an indicator of good design. but hear it is known as being poorly designed!

thanks for any clarification

Same distribution yes, but should include all possible scenarios of occurence, so its not concentrated in a just a few examples i.e. it should include hopefully the entire distribution.

thanks gent.spah.

so it might be better to say that the original data doesn’t span all the distribution rather than saying devset is poorly designed. agree?

if we wanted to have a dev set which has a good design, how should it be?

We could say both depending on where the issue is, a good dev set should normally be of similar distribution as training set but that doesn’t mean that your test it in real life it might perform well! Why, because the train set doesn’t include the entire distribution.