The video talks about using the training, evaluation, and test set to decide what model to use. I don’t understand why we have to use an evaluation set to compute the cost and decide based on the cost, while we can just compute the cost using the test set instead. All of them have different data so computing the cost using the test set should not be a problem.
My understanding is this:
first, you train different models using the training set
Then, you compute the cost using the trained models and the evaluation set as input. Then, you compare the cost value.
If it is correct, why can’t we just use the test set instead? Why even need all 3? The video tried to answer that question but I don’t really understand. Can anyone please help me?
thank you
@wiln you never want to look at (or touch) your test set because you risk biasing your entire model. The whole idea is building an effective model of prediction on unseen outcomes-- Otherwise you risk overfitting everything.
I agree with @Nevermnd.
Here’s how I look at it:
- The training set is used for training.
- The validation set is used for fine-tuning the model.
- The test set is used only once, to verify that your completed model gives “good enough” performance.
You said the validation set is used to fine-tune the model. Does that mean the validation set is used during training and not after training? The example in the video makes me think that the validation set is used after training the model to compare the costs and choose the best model.
It’s an iterative cycle.
- You train the model.
- You run the validation set through the model and look at the results.
- You change some parameters for the model and go back to 1).
Eventually (when you’re not making any further improvements) you stop the loop and verify the model using the Test set.
I had the same question as you posted.
I checked the responses to your question and some of them makes sense, but to your point the example in the lecture had no need of test set.
I am yet to have a convincing answer as to why/how can the test set be overly optimistic error for the model.
We checked the error on an unseen (validation) dataset, which is the same as checking error on a test dataset.
For Fine-tuning/ hyperparameter tuning as well there is no need for a separate test set as the model never gets trained on validation set and for it (even after tuning) it is a new/general dataset.
Happy to discuss more on this as I might be missing something here.
The validation set is used for adjusting the model (i.e. setting the amount of regularization to avoid overfitting).
Then you use a test set to verify the final performance.
I have a subsequent question to this: Once I’m done with model selection using the validation set and having verified the model results using the test set, if I now wanted to ship this model to production, would I train it on the complete dataset in order to give the model as much data to learn from as possible? Or would I still use only the training set from the model selection stage to train the model?
No, do not retrain after testing.