Train/Dev/Test Distributions Lecture Clarification

Hi Sir,

@paulinpaloalto @lucaswener @Vasanth @carloshvp @ashimono @Sabs @manifest @fginac @Jendoubi

Im not able to understand the context “consider important to do well on” what does it meant ? can u please help to understand ?

choose a dev set and test set to reflect data you expect to get in future and consider important to do well on.

we can able to understand the point choose a dev set and test set to reflect data you expect to get in future but consider important to do well on …meaning what sir ?

Hi,

When you are building the model, you do not ideally have access to the test data. You only have access to the dev set or the validation data. Thus you might want to choose the model which has the best accuracy/ loss on the dev set. However, the distribution of the dev set might be very different from the test set. So you would get very bad accuracy on the test set; something which you do not want.

On the other hand, you might have generated a validation set which is exactly equal to the test set and just left it at that. You did not try to improve the score on the dev set which leads to a low score on the dev as well as test set.

Ideally, you would want a good score on the dev set and the dev set should be a good representation of the test set.

Hope this helps!

1 Like

Sir As a conclusion we can say that consider important to do well on means, we need to make best accuracy results on the dev set right ?

Sometimes accuracy is not the best metric. But yes, you should choose the model weights which optimizes the metric you want!