In the Step 4 - Train the Model
I changed the EPOCHS to 40, to see how the model would improve.
I ended up getting this where the model improved for awhile, then got really messed up and had to relearn and ended up only slightly better then it started.
I was curious what would cause such a massive drop in accuracy?
2nd attempt training with 40 Epochs:
3rd attempt training:
The cost surfaces are in very high dimensions and are incredibly complex. You can “go off a cliff” at any point in the training. It might be worth trying a different optimization algorithm that uses more sophisticated controls on learning rate and the like. Here’s a thread which talks a bit more about non-convexity and also links to a paper by Yann LeCun’s group that is worth a look for insight about whether reasonable solutions exist and can be found or not.
The other point here is that there is some randomness in TF/Keras training, meaning that you don’t get the same results every time, even if you try to set random seeds and control the randomness. I’ve been meaning to research the TF docs more to see if this is controllable, but haven’t gotten around to it yet.
A useful question is:
Are those changes in accuracy even meaningful when the number of iterations is so low?
Here’s tensorflow 2.11 api for better reproducibility across the same hardware.