Train error is useless

Hi I just finished the part about advice for applying machine learning. I believe that I understand the procedure in choosing a model:

  1. Train the different architecture using your training set.
  2. After finishing your training, calculating the corresponding cross validation error and test error (generalization error termed by lecture).
  3. Choose the model with lowest cross validation error.
  4. Report the generalization error (test error).
    It seems that the train error is useless in the whole process. Am I right?

Hi @empheart

It depends on what you call “useless”. It definitely indicates you many things , one for example, you can measure how much your model overfits (comparing it with cross validation error, if you had not had training loss - how would you know). The rate it goes down also is informative. You can compare it with pure chance probability - how much are you improving. And there are many other things.

But if you are purely talking about how much value will your model bring to the end product, then training error is not that important (test error is the one you would be looking at).

I guess what I’m saying is that train error is important (a lot) in your “1.” point.

3 Likes