I try to figure out the scope of the different methods listed above.
My understanding is, that custom training will replace:
- model.fit() : reducing loss, adjusting weight, calculating Gradients
- model.compile() : defining loss, metrics and optimizer
But I am still at loss in regards of the initialization of the weights of the model.
Previously I understood that the initial weights of a model will be defined already once the model constructor model.init() is called, which is calling model.build()
(At least this is what I understood from the Course in Custom layers and models)
To do so, the model constructor already need to know the shape of the inputs.
But in the multi-GPU mirrored strategy lab, I do not see where the Input-shape of the model
When will the model be initialized and where to specify the input shape?
And will each model (since every GPU has its own copy) have the same set of initial weights?
Can we choose, how to initialize the weights (since there are different strategies?)