In the explanatory text of this Lab, it is stated that the encoder part outputs the mu (mean) and sigma (standard deviation) of the latent representation. However, in fact it outputs the natural logarithm of the variance instead of the standard deviation. Thus what is called sigma in the code, is actually log(sigma**2) if we use the standard notation where sigma is the standard deviation. This is immediately obvious from both the computation in the Sampling
class and the KL-divergence part of the loss function kl_reconstruction_loss()
.
It would be great if this could be corrected in order to prevent possible confusion for learners.
I also have a question about this: does it have any advantage to predict the log variance wrt predicting the sigma with the encoder? If so, there could be a sentence or two about this in the Lab, or at least a relevant link.