Hey @starboy, When I was taking the course, I had a similar query, and I discovered a credible explanation that satisfied me. But now that we have this platform and a wonderful community to assist us in resolving our concerns, I believe you and I will be able to receive a better answer. So, don’t stop with my explanation; keep looking.
My understanding is something like this:
Controllable Generation is where we tweak the noise vector in order to get output with different features, right? This is accomplished by using a basic stochastic gradient ascent to build a noise vector that creates more of a certain characteristic, such as a noise vector that solely modifies hair colour or hairstyle. That can be seen directly after the code snippet you supplied in your query (see screenshot below).
Now, where does it happen? Where do we find this actually happening in the training timeline? According to Sharon (the instructor) in the Course 1, Week 4, 4th Video and I quote:
Conditional generation, allows you to specify what class you want to a very different type of thing. For this, you need to have a label data set and implemented during training typically. You probably don’t want to label every hair length value, so, controllable generation will do that for you and it’s more about finding directions of the features you want. That can happen after training. Of course, controllable generation, you will sometimes also see it happening during training as well. To help nudge the model in a direction where it’s easier to control. Finally, as you just learned, controllable generation works by tweaking that input noise vector Z that’s fed into the generator, while with conditional generation, you have to pass additional information representing the class that you want appended to that noise vector.
So, what do we mean by this? We mean that this adjusting may be done both after and during training.
My understanding ends here; I hope someone can explain it more thoroughly than I can since I, too, am curious about what and how it happens behind the hood.
Hope it helps