The consequence of different distribution in train dev and test

Course Q&A Deep Learning Specialization Structuring Machine Learning Projects
2

Hey @Sara,

I love your questions :slight_smile:
These two will be easy to explain though.

We train our model on the training set and evaluate the model on dev and test sets. In a sense, the purpose of the test set is to make sure that our evaluation on the dev set is correct (we expect both dev and test errors to have close values).

The purpose of dev and test sets to perform evaluation on them. If they had different distributions we wouldn’t be able to compare errors and reason about results.

The true data distribution is unknown. We only have access to empirical data distribution – our training set.

  • If we evaluate our optimization of model parameters using data that came from the same distribution as our training set, we will be doing our best (we aim almost to the true target).
  • If we evaluate our optimization using data that came from a different distribution, we may still get some decent results, but they will be worse than with the previous option (we aim further from the true target).
2 Likes