About dev and test sets

The test set should only be used once as a litmus test before deploying.

Usually you need the val set because architecture, hyperparameters or or features are adjusted and improved just because CRISP-DM is a highly iterative process.

Cross-validation is a nice approach to avoid overfitting and see how well the model generalizes. Also: This thread might be worth a look for you, @abdou_brk: How and why do training and cross validations sets wear out in time? - #5 by Christian_Simonis

Hope that helps!

Best regards
Christian

1 Like