Would appreciate if someone could share how the two magic number in the test_gen_loss() function was derived.
# check that the loss is reasonable
assert (gen_loss - 0.7).abs() < 0.1
Which magic numbers?
I would guess that the author of the assignment determined that those are useful thresholds to determine if your code works correctly for that test case.
Yes as @TMosh says, they would have tried it earlier. Just Reverse-Engineer it, for the given z_dim, num_images and other parameters they would have got a gen_loss value close to 0.7, and by testing edge cases they would have derived the threshold because the noise need not be the same all the time, so it need not be strictly 0.7 which is why they would have kept a threshold for the difference. Makes sense?
I’m talking about 0.7 and 0.1 here.
As I said before, those values were experimentally determined by the author of the assignment. They appear to be sufficient to verify that your code works correctly.
Add some print statements to show the values that you actually get when you run the test multiple times. Note that they don’t seem to set the random seeds to make the results repeatable, so (as Tom and Nitin have said) they must have a strong statistical evidence for those “magic” numbers.
Agree. If there really is some “strong statistical evidence”, then the authors possess really interesting insights about GAN.
Or, they just tried it, found that it worked well enough, and went on to their next task.
The folks who built the GANs courses are very experienced programmers and know their subject well. Maybe you could argue their test code is too clever by half, but it is definitely at a different level than the corresponding code in DLS. Take a look at the function test_gen_reasonable and watch the tricks they play with the fact that several of the arguments to your get_gen_loss function are functions. Watch what they pass as the function arguments and how they handle the results.
On the simpler test_gen_loss function, they use the “real” gen, disc and criterion functions, but they are taking advantage of the fact that no training has been done for the Generator and the Discriminator, so the parameters are all initialized with the Uniform distribution (that’s the PyTorch default). I don’t claim to understand the statistics at work here, but I did what I suggested in my earlier post and just added a print statement to show the gen_loss value and ran test_gen_loss 4 times. Here are the values I got:
gen_loss 0.7121910452842712
gen_loss 0.7054029107093811
gen_loss 0.6311353445053101
gen_loss 0.6718310117721558
Note that the values do bounce around, so they really aren’t setting seeds to make things reproducible. But the values are all within 0.1 of 0.7. Hmmmm. Do you want to do the statistics to figure this out or just assume that it’s not an accident and spend your mental energy on the actual course materials instead? Your choice 
…
3 Likes
You’re right. Very likely the authors took advantage of Pytorch’s default way of parameter initialisation to get the bound of the loss.
Thank you.