Yes, I have tried running a lot of experiments and I get varying behavior from the training here. My first thought was that they don’t set any random seeds anywhere in this notebook, so maybe that’s what causes the variability. But if I add:
tf.random.set_seed(42)
Either early in the notebook or right before the instantiation of the model, I still get varying results from the training. I read through the documentation about TF “global” versus “operational” seeds and my belief based on those documents is that setting the global seed as above should give reproducible results.
I don’t have enough samples yet to really conclude that the degree of variability is less with the seed fixed, but that is the impression I get. But it still definitely varies.
I have not tried printing the initialized weights to make sure my seed logic actually does produce the same results, but that is my reading of this TF docpage.
So my next theory is that model.fit with Adam optimization has its own source of randomness that is not affected by setting the seed. But I have not had time to pursue that. The next step is to read carefully the documentation on model.compile and model.fit.
The solution surfaces of Neural Networks are incredibly complicated, so there is never any guarantee that you’ll get smooth and monotonic convergence from Gradient Descent: you can go off a cliff at any iteration, especially with a fixed learning rate. But the internal algorithms that TF uses in model.fit presumably use sophisticated techniques for modulating the learning rate based in the magnitudes of the gradients, but the evidence seems to show that even with state of the art sophistication you can still get weird behavior on any given training run.