TF batch norm for CNNs question

Course Q&A Deep Learning Specialization Convolutional Neural Networks
1

Hi,

Having trouble finding information on this topic…

Why is the axis = 3 when we do batch norm? I.e., why are we doing batch norm over the channels axis. When would we want to do batch norm over another axis instead?

In the documentation (linked below), they demonstrate it using axis=-1 and note:
" For instance, after a Conv2D layer with data_format=“channels_first” , set axis=1 in BatchNormalization."

Thanks!

2

The normal orientation of the tensors we are dealing with here (for images) is:

samples x height x width x channels

So axis = 3 corresponds to channels, but note that axis = -1 in python means “use the last axis”, which is another way to say 3 in this case.

If you use “channels first” orientation, then the dimensions are:

samples x channels x height x width

which is why you would need axis = 1 in the latter case if you want to normalize across the channels.

1 Like
3

As to why it is channels, I do not really know. But if you think about it, there is a symmetry between the vertical and horizontal pixel axes, right? Why would you normalize on the height dimension versus the width dimension? But channels express what is going on at a given pixel. Note that once you are beyond the input layer of a ConvNet, the tensors are not really “images” any more, so calling them “pixels” may be a little misleading, but there still is the geometry of height and width which may reduce as you go through the layers. The channel values no longer represent RGB color values but are just real numbers that represent some aspect of what the network is detecting in the images and of course there can be lots more channels than 3 in the internal layers of a deep ConvNet.

1 Like
4

My understanding is to just take a channel like it is a feature, and in fact, sometimes people call a channel as a feature map. In a tabulated dataset, when we normalize, we normalize along each of the features, right? We are just doing the same to images and normalize along each of the feature maps.

Welcome to the community, @whthomps

Cheers,
Raymond

1 Like
5

Thanks so much! Appreciate the simple explanation.

6

Thank you very much for the context - this makes a lot of sense.