The video talks about using the training, evaluation, and test set to decide what model to use. I don’t understand why we have to use an evaluation set to compute the cost and decide based on the cost, while we can just compute the cost using the test set instead. All of them have different data so computing the cost using the test set should not be a problem.
My understanding is this:
first, you train different models using the training set
Then, you compute the cost using the trained models and the evaluation set as input. Then, you compare the cost value.
If it is correct, why can’t we just use the test set instead? Why even need all 3? The video tried to answer that question but I don’t really understand. Can anyone please help me?
thank you
@wiln you never want to look at (or touch) your test set because you risk biasing your entire model. The whole idea is building an effective model of prediction on unseen outcomes-- Otherwise you risk overfitting everything.
2 Likes
I agree with @Nevermnd.
Here’s how I look at it:
- The training set is used for training.
- The validation set is used for fine-tuning the model.
- The test set is used only once, to verify that your completed model gives “good enough” performance.
2 Likes
You said the validation set is used to fine-tune the model. Does that mean the validation set is used during training and not after training? The example in the video makes me think that the validation set is used after training the model to compare the costs and choose the best model.
It’s an iterative cycle.
- You train the model.
- You run the validation set through the model and look at the results.
- You change some parameters for the model and go back to 1).
Eventually (when you’re not making any further improvements) you stop the loop and verify the model using the Test set.