In the week 2 Alpaca transfer learning, it strikes me as curious that throughout training, performance on the cross-validation data is consistently better than on training data by both accuracy and cross-entropy. I’m not used to seeing that behavior. Would anyone care to comment? Is this a result of augmenting the training but not the validation set?
Thanks!
When images in cross-validation set are a lot closer to the images the model predicts correctly in the training set, your results can be observed.
The reason for augmenting images is to get around overfitting the training set.
One way for you to explore further is to remove ImageAugmentation parameters other than rescale and check the behavior.
The other point to make is that the training here is non-deterministic. They don’t set any random seeds here, but I tried that and still get different results everytime. FWIW I don’t not see the case that the validation accuracy is > training accuracy in the experiments I tried. Here’s the training before fine tuning:
Epoch 1/5
9/9 [==============================] - 9s 956ms/step - loss: 0.7295 - accuracy: 0.5115 - val_loss: 0.6816 - val_accuracy: 0.3846
Epoch 2/5
9/9 [==============================] - 7s 812ms/step - loss: 0.7374 - accuracy: 0.4580 - val_loss: 0.6930 - val_accuracy: 0.3846
Epoch 3/5
9/9 [==============================] - 8s 844ms/step - loss: 0.7246 - accuracy: 0.4427 - val_loss: 0.6685 - val_accuracy: 0.3692
Epoch 4/5
9/9 [==============================] - 7s 823ms/step - loss: 0.7139 - accuracy: 0.4351 - val_loss: 0.6903 - val_accuracy: 0.3846
Epoch 5/5
9/9 [==============================] - 7s 812ms/step - loss: 0.7005 - accuracy: 0.4466 - val_loss: 0.6688 - val_accuracy: 0.3846
And here is the fine tuning run:
Epoch 5/10
9/9 [==============================] - 10s 1s/step - loss: 0.8221 - accuracy: 0.4275 - val_loss: 0.6876 - val_accuracy: 0.3846
Epoch 6/10
9/9 [==============================] - 9s 1s/step - loss: 0.6995 - accuracy: 0.5115 - val_loss: 0.6655 - val_accuracy: 0.6154
Epoch 7/10
9/9 [==============================] - 9s 968ms/step - loss: 0.6991 - accuracy: 0.4847 - val_loss: 0.6722 - val_accuracy: 0.3846
Epoch 8/10
9/9 [==============================] - 9s 989ms/step - loss: 0.6957 - accuracy: 0.4924 - val_loss: 0.6817 - val_accuracy: 0.3846
Epoch 9/10
9/9 [==============================] - 9s 968ms/step - loss: 0.7252 - accuracy: 0.4122 - val_loss: 0.6675 - val_accuracy: 0.3846
Epoch 10/10
9/9 [==============================] - 9s 1s/step - loss: 0.7183 - accuracy: 0.4695 - val_loss: 0.6816 - val_accuracy: 0.3846
Actually now that I look at those, it’s a little suspicious: the validation accuracy is mostly constant at 0.3846 and the training accuracy bounces around (not monotonically increasing). Hmmmm. I would venture that this is not typical behavior and that further investigation and perhaps hyperparameter tuning is warranted here.
Hi Paul,
Hope you’re well. Did you by any chance reply to me by misake?
I haven’t been active on the Discourse group or posted anything recently.
Regards,
-jh-
Sorry, but they have asked the mentors to search for unanswered questions and make sure they get resolved to make the forum stats look better. Your question is a good one and I apologize that no-one responded when you first asked it. There is still value in answering old questions, since the history on the forums has ongoing value. Unlike the Coursera forums (the search engine of which is disfunctional), people are able to find pre-existing posts on Discourse and derive value from them.
Regards,
Paul
Thanks, Paul! No problem. I just wanted to make sure you hadn’t accidentally replied to the wrong person, leaving a live user’s question unanswered.
Regards,
-jh-