I dont understand, can someone give me a simpler intuition on why we cannot use Jtest as a measurement of how a model is doing? Is it because, if we only focus Jtest to be small, then it will overfit to only the test dataset?
Furthermore, it is mentioned that to pick the best model, we need to have a low Jtrain and Jcv. Why dont we just use the lowest Jtrain and Jtest?
Hi @Zolids
If you tune your model to minimize J_{\text{test}} , you risk overfitting to the test set. The test set is meant to be a final, unbiased evaluation of your model’s performance on unseen data, not a part of the training or model selection process.
Th reason we use low J_{\text{train}} and J_{\text{cv}} is because J_{\text{train}} ensures the model fits the training data well and J_{\text{cv}} gives an estimate of the model’s performance on unseen data (to avoid overfitting).
Hope it helps! Feel free to ask if you nees further assistance.
So basically Jcv and Jtrain data should come from the same distribution, however Jcv is only used to check which models is best to use, and Jtest is only for the final evaluation?
Do we fit the parameter using Jcv too?
Thanks for answering anyways
Yes, J_{\text{train}} and J_{\text{cv}} should come from the same distribution, with J_{\text{cv}} used for model selection and J_{\text{test}} reserved for final evaluation. We fit the model parameters using the training set and use CV to select the best model by comparing J_{\text{cv}} across different models. The test set is only used once at the end to provide an unbiased estimate of the model’s performance on unseen data.
