Hello Folks,
After finishing the last assignment of Convolution Networks Week 4, I started to play with the program, changing images and Hyperparameters on Pycharm.
Unfortunatelly, when the generated image is loaded with a noisy version of Content image, right on the 200th epoch, it starts to become like a very noisy blur with a grid pattern and from this point never converges back so any resemblance with the content image. I´ve tryed altering the percentage of influence of style image on the layers, the Alpha/Beta, and also reducing the learning rate with no luck. Also tryed to raise the epochs to 25000, but with no success as well. Any hints about what should be happening and how to avoid it ?
Thanks a lot friends!
Regards,
Ricardo
I don’t understand what you mean by that. The generated image is an output, right? It’s not the input to anything. Do you mean you started with a noisy version of the Content Image?
The first thing to check for is “versionitis” problems. If you are taking the code from any of the course notebooks (especially those using TF/Keras) and running them in your own environment or on Colab, there is a whole field of landmines to step on there. The course material uses versions of TF and the various packages that are > a year old at this point. Things change in this space pretty quickly. Here’s a thread which discusses some ways to duplicate the environment.
The other general point here is that the way this algorithm works is that the more iterations you run, the further you are getting away from the original content image, right? So I wouldn’t expect it to “converge” more in the direction of the original content image if you run more iterations. If it’s already weird, running more iterations will make it weirder. This “style transfer” assignment is sort of different than a lot of the others in that there is not a defined “correct answer” that we’re trying to optimize toward. This is all just a matter of aesthetics: what do you think looks cool? And the answer may be different for different people.
The other high level point is that the results may matter a lot depending on whether you are training from scratch every time or whether your are continuing the training based on the previous training you have already done. Of course “from scratch” in this context means starting from the pre-trained VGG weights as the starting point.
None of the above is really an “answer”, but just some suggestions for further conversation/investigation.
Hi Paul,
Sorry for the shallow explanation, but It revolved around the assignment. On the exercise we were supposed to pick a content image and a style image and perform an image style transfer using a pretrained Vgg network. To apply the image style transfer, we were supposed to initiate the generated image with the content image + (tf.add) some noise: (tf.random.uniform(tf.shape(generated_image), -0.25, 0.25)).
Reading your considerations, I agree that it can be an environment issue. The model is pretrained, and on the Colab the result came pretty neat. But after changing to my local environment (with later versions of tensoflow and keras), the image becomes a colored version of random pixels like a TV out of tune.
I agree also that there´s no “correct answer”, but the result is far away from either content or style images.
Thanks for the time and attention.
1 Like
If the results just look scrambled, are you sure it’s not just a “rendering” problem or something of that nature? Which could be a “versionitis” issue …
Yes, I´ll take a look at the thread you´ve mentioned.
Thanks!
This is interesting, since I’m using Tensorflow 2.9.1 in my local system. I’m using exactly same Jupyter notebook with given libraries downloaded from Coursera platform. Of course, some assignments need Tensorflow 2.3 or 2.4 with additional libraries associated to those, but this particular assignment works on 2.9.1.
Can you paste two images ? One is an image just after a noise injection, and the other is an image created at 200th epoch if possible. Do not need to hurry as I will not be on-line for next 6-8 hours…
Hi Nobu!
Of Course!
See below the link for the first initiation image and after the 250 epoch.
Microsoft OneDrive - Access files anywhere. Create docs with free Office Online.
Additionally, I suspect some exploding gradient issue, because I recieved the following warning:
RuntimeWarning: overflow encountered in long_scalars
Somehow getting overflow
Does this make sense?
Thanks!
One more thing:
My Tensorflow is 2.8 and Python 3.9
Regards,
At first, using your image data (without adding extra noise), I tried to run NST on my local Jupyter notebook. It worked pretty well.
Then, I quickly ported code to my Pycharm environment, just like you did, and made it run. Here is the result at 2500 epoch. Looks like your image fits to a Monet style quite well. 
To apply the image style transfer, we were supposed to initiate the generated image with the content image + (tf.add) some noise: (tf.random.uniform(tf.shape(generated_image), -0.25, 0.25)).
In addition, we need to cast back from EagerTensor to tf.Variable. I suppose you did already, but for other learners who may come to here. In our assignment, this important instruction is part of test cell, which is easily forgotten.
generated_image = tf.Variable(generated_image)
My environment is tensorflow-macos 2.9.2/tensorflow-metal 0.5.0, and Python 3.8.12.
Assuming that you completed this assignment already, I can send my Python file in Pycharm environment for your reference.
RuntimeWarning: overflow encountered in long_scalars
Yes, this is a big problem in your environment, I suppose.
Hello Nobu!
Seing your “correctly” generated image is a relief ! Yes I would like to see your Python file from Pycharm to validate any misalignments.
Somehow, even with the default hyperparameters, this overflow warning seems to be the cause of the error.
Thanks again for the help!
Ricardo,
I sent my file privately. Please post any findings that can be shared with other audiences, who may come to here, thank you !