Hello,
I finished the WGAN lab, and noticed that after running the final training cell for a while, the loss of the generator eventually started increasing while the loss of the critic stayed very low.
The image is what the lab produced after training for 50 epochs. You can see that the losses appear to be nearly parallel at first, but then the generator loss starts increasing around the 500th display step while the critic loss remains roughly the same throughout training.
This makes me wonder if the generator is failing to learn, and more broadly, it made me wonder how to actually know if the generator is improving. If there is only a single network training on static set of images, the single loss tells us clearly whether the network is improving. But since the generator’s loss is relative to the skill of the critic which is also changing, it seems plausible that the generator could improve while nevertheless having an increased loss.
How do we know whether our generator is actually improving (other than looking at the output and making a rough guess)?
We measure the performance of the discriminator, if it fails detecting a large percentage of generated fakes then it means the generator has good skills in fooling it. Thats what is basically done out of a batch you measure how many are detected as fake and those reduce the generator ‘accuracy’/skill. Of course the discriminator and generator need be continuously trainning, a loss reduction on discriminator means its getting better.
if it fails detecting a large percentage of generated fakes then it means the generator has good skills in fooling it.
But it could equally mean that the discriminator is just bad, not necessarily that the generator is good. If the generator and the discriminator have an average loss of 0.5 at the end of the second epoch and then they again have an average loss of 0.5 at the end of the 500th epoch, that doesn’t mean that they didn’t improve. They may have failed to improve, or they may have improved equally. Both cases would result in the loss being 0.5, wouldn’t they? So is there an absolute measure of a generator’s ability?
You assumptions make sense, i dont remember the specialization fully now, but I think that you will get to know a measure that does that eventually, if not already.