Does skip connections increase the number of channels?

I was re-watching the U-Net architecture video in week 3 of the CNN course. At around 4:40 Prof. Ng says that the skip connection is copied over from the left to right and then goes on to show an increased number of channels by the skip connection. But in the Resnet video of week two only activations were added to the Z values in the later layers. In the U-Net architecture video, the thickness refers to the number of channels. So why does the thickness of the layers increase after adding the skip connection?

Hi again @Shivam_sharma_MED_07 :slight_smile:

I have just re-watched as well the section of video that you mention. I am not sure where Prof. Ng mentions that the skip connection increases the number of channels. What I hear is that we add the skip connection to the Trans Conv. As you can see in the video and also in the lab’s code, the operation between the Skip Connection and the Trans Conv is a concatenation (keras.Concatenate). This keras function includes an ‘axis’ argument which indicates the concatenation axis. The final effect of the concatenation will depend on the axis argument passed to it.

It could happen that concatenating two tensors on a certain axis, could increase the number of channels, or could increase its Height or its Width, so it depends on how the concatenation is done.

Lets see some examples:

Lets take two tensors, say 2 images, of size 100x100 with 3 channels.
input1 = tf.keras.layers.Input(shape=(100, 100, 3))
input2 = tf.keras.layers.Input(shape=(100, 100, 3))

Now lets concatenate them on axis = 1:
concatenated = tf.keras.layers.concatenate([input1, input2], axis=1)
Result: KerasTensor(type_spec=TensorSpec(shape=(None, 200, 100, 3)…
See how this case increases the height.

Now lets concatenate them on axis = 2:
concatenated = tf.keras.layers.concatenate([input1, input2], axis=1)
Result: KerasTensor(type_spec=TensorSpec(shape=(None, 100, 200, 3)…
See how this case increases the width.

Now lets concatenate them on axis = 3:
concatenated = tf.keras.layers.concatenate([input1, input2], axis=1)
Result: KerasTensor(type_spec=TensorSpec(shape=(None, 100, 100, 6)…
See how this case increases the channels.

So it all depends on the specific implementation (shape of tensors, axis of concatenation).

Thoughts?

Juan

But adding a skip connection only means adding activation values from a layer on the left to the Z values of a layer on the right which has the same size (w,h,c). So skip connection should be an addition operation only, right? and not a concatenation operation. And skip connection should not increase w,h, and c. In the resnet video in week 2 at 3:00 Prof. Ng mentions only addition and no change in size and shape due to skip connection.

So basically I have two questions.
Does skip connection change the w,h,c of a layer?
Isn’t skip connection only addition as explained in the lecture but not a concatenation of matrices?

Hi @Shivam_sharma_MED_07 ,

Valid questions… so I went ahead and did this to experiment and be able to answer with facts :slight_smile:

In the upsampling function on the lab of U-Net, I added some print statements:
print(‘Input1:’)
print(up.shape)

print('Input2:')
print(contractive_input.shape)

print('Concatenation:')
print(merge.shape)

I am adding these print lines right after the concatenation of the skip connection with the corresponding TransConv.

Then I executed the model… downsampling… and then upsampling. This is the result I got:

Input1
(None, 12, 16, 256)
Input2
(None, 12, 16, 256)
Concatenation
(None, 12, 16, 512)

Input1
(None, 24, 32, 128)
Input2
(None, 24, 32, 128)
Concatenation
(None, 24, 32, 256)

Input1
(None, 48, 64, 64)
Input2
(None, 48, 64, 64)
Concatenation
(None, 48, 64, 128)

Input1
(None, 96, 128, 32)
Input2
(None, 96, 128, 32)
Concatenation
(None, 96, 128, 64)

As we can see here, it is effectively altering the channels of the intermediate results, but we cal also see that it goes from 512 channels in the first skip connection, down to 64 in the last one, before the last layers.

Now your questions:
Q)Does skip connection change the w,h,c of a layer?
A) Yes, in the U-Net the concatenation is altering the channels.

Q)Isn’t skip connection only addition as explained in the lecture but not a concatenation of matrices?
A) As we can see, in U-Net is concatenation. I have seen other skip connections in other models where it is added, for example in ResNet, it is added. In DenseNet it is concatenated.

Thoughts?

Juan

1 Like

Thank you so much for your answer. Now the concept is more clear.

1 Like