Problems regarding normalization of image dataset

Course Q&A Generative Adversarial Networks (GANS) Apply Generative Adversarial Networks
1

In the cycle-gan code, when author do the operation (item_A-0.5)*2 , what exactly of this do?
if dataset already normalize to range [0,1] it will shift the midpoint of that image to 0, and shift the range of that image to [-1,1],right? but original image is range from 0~255,right? So what is this operation do?

2

Hi, @gemini_vi!

Firstly, transforms.ToTensor() will scale image pixels from [0,255] to [0,1] values. Afterwards, (item_A-0.5)*2 maps from [0,1] to [-1,1]. The reason of re-mapping from [0,1] to [-1,1] is related to the choice of activation function in the architecture (in our case Generator). Tanh activation produces outputs in the range [-1,1]. So for the consistency and computational stability of the training the input tensor is normalised to [-1,1].
To be clear, this doesn’t mean that normalising an image to [0,1] is wrong. Theoretically, the training should still work without re-mapping to [-1,1]. The choice of the normalisation range is determined by the choice of the activation function.

References:

Kind regards,
DK.

1 Like
3

thank you very much! got it, and another question regarding of weights_init function, author used if isinstance(m,nn.BatchNorm2d) while in both ContractingBlock and ExpandingBlock only instancenorm were used, so this should be isinstance(m,nn.InstanceNorm2d),right?

4

I think there’s a slight confusion - weights_init() is a method to initialise the weights of a layer. Depending on the layer type (BatchNorm2d or Conv2d) different initialisation techniques are used.
This is not to be confused with instance initialisation InstanceNorm2d. InstanceNorm2d is very similar to batch normalisation (except mean and variance are not computed over the batch dimension). To conclude, the purpose of weights_init() is very different from self. instancenorm in ContractingBlock and ExpandingBlock.
Please let me know if I haven’t understood the question.

Kind regards,
DK.

Update: You can think of weight_init() as something you do once when you create a layer. In contrast, instance normalisation is applied on a batch basis.

1 Like
5

Thank you for your reply, I am new to DL, please don’t mind if I have some stupid question. There is definitely a confusion here, there is no batchnorm2d layer throughtout whole model, so why initialize weight for batchnorm2d, instead of using batchnorm2d author used instancenorm2d layer,right? so we don’t need to initialize weight for instancenorm layer?

6

hi, @gemini_vi, your questions make sense and I had to do a bit of research myself to refresh my memory, so we’re learning together!

Firstly, weights_init is applied recursively to every submodule of Generator class.
From pytorch documentation:
apply(fn ) fn is (weights_init()) in our case.

Applies fn recursively to every submodule (as returned by .children()) as well as self. Typical use includes initializing the parameters of a model (see also torch.nn.init). Note, to every submodule of Generator class (including contracting and expanding blocks).

Secondly, regarding the weight initialisation of InstanceNorm2d.

Screenshot 2023-09-11 at 17.18.16
Screenshot 2023-09-11 at 17.18.161610×342 38.8 KB

In the formula above also from Pytorch documentation, please note that when affine is True an instance normalisation layer has 2 learning parameters - gamma and beta. When affine=False (by default) InstanceNorm2d has no learnable parameters. Hence, no initialisation of weights is needed.

Thirdly, I’ve done some experiments.

  1. affine=False. This snippet throws an exception

AttributeError: ‘NoneType’ object has no attribute ‘normal_’
m.weights in NoneType because there’re no learnable parameters!

m = nn.InstanceNorm2d(3)
print(isinstance(m, nn.BatchNorm2d))
torch.nn.init.normal_(m.weight, 0.0, 0.02)
torch.nn.init.constant_(m.bias, 0)
print (m.weight)
  1. Setting affine=True
m = nn.InstanceNorm2d(3, affine=True)
torch.nn.init.normal_(m.weight, 0.0, 0.02)
torch.nn.init.constant_(m.bias, 0)
print (m.weight)

output:

InstanceNorm2d(3, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)
False
Parameter containing:
tensor([-0.0081, -0.0119, 0.0036], requires_grad=True)

To conclude, indeed the code below is never executed.

  if isinstance(m, nn.BatchNorm2d):
      torch.nn.init.normal_(m.weight, 0.0, 0.02)
      torch.nn.init.constant_(m.bias, 0)

And no, we don’t need to initialise the weights of InstanceNorm2d when affine is set to False.

Kind regards,
DK.

7

Thank you very much, Got it!