Hi everyone,
I just read an article about VAEs and i can see that after go through the encoder, we will have means and standard deviations, then it will pass through two path, one for KL loss and one to compute reconstruction loss.
My question is, if we have two or more path to compute, then how we can update the variables ? In this case, KL loss and retcon loss have different space (or norm) to measure so we can just average it or sum it up cause too big.
1 Like
Please see Define the VAE as a Model with a custom train_step
section.
2 Likes
The two KL and Reconstruction losses are combined into a single loss. Since the losses are in ndifferent spaces, most implementations weight the KL term using a hyperparameter, B. During training the gradients of the combined losses with respect to the network parameters are calculated using backpropagation. The optimizer uses this gradients to update the parameters in a way that minimizes the total loss…
1 Like