Week 1: Train / Dev / Test video

Dear Mentors,

In the train/dev/test video at 9:10, Mr Andrew says

Make sure the dev/test set comes from the same distribution.

I understand that if the algorithm performs well on the dev set during the training phase. Then, After we select our best model, the algorithms will perform well on the test set as well if the distribution of dev and test same it the same.

But, I wonder why he doesn’t mention anything about train and dev set distribution. I guess if train and dev set distribution is not the same/similar, we will always have a false notion that our model is well trained even if it performs well on that particular dev set. It might not generalize well. Even if it performs well on that particular dev set. It is possible that it doesn’t generalize on the test set despite having the same distribution of test and dev set? or It starts performing badly on real data.

Like Is it possible that the train and dev set have somewhat different distributions ‘say both normal but with different mean or same mean but different variances’? Do we consider this kinda data for training?

Note: Usually, I carve a dev set out of training data only. I understand it will have a similar distribution in that case, but I don’t know if it is always the practice that incorporates research as well.

How do you decide and these train/dev/test in practice? It would be great if you throw some light here or refer me to papers or blogs for its strategy.

I have studied course 3. I did not get an answer there as well.

1 Like

They should all come from the same distribution.

Ideally, you have one big pile of data, and you shuffle it up and split it into three smaller piles.

1 Like

Can you refer to some papers or blogs you have read on this topic if possible, Please?

1 Like

Sorry, I have no such references.

1 Like

Sorry, but I think Prof Ng does cover all this in DLS C3. It’s been a while since I went through the lectures there, but in my notes there are several lectures in Week 1 that are relevant, but the real material with the most detail on that point is in Week 2. E.g. start with this lecture and watch the next 2 after that as well.