Skipping in resnets

Course Q&A Deep Learning Specialization Improving Deep Neural Networks: Hyperparameter tun
1

I don’t under how a[l+1] is really skipped in this example:

image
image971×183 30.3 KB

In the example above we go straight from a[l] to a[l+2] thus skipping a[l+1].
but as shown on the formula below,
image
image1021×208 88.7 KB

in order to calculate a[l+2] we do have to pass through the calculation of a[l+1].
So, it seems to me as if we are actually not skipping the layer l+1 after all. In what way are we skipping l+1?

2

Actually, “a[l]” skips a[l+1] layer, But in the algorithm for calculating a[l+2] we use both a[l+1] and a[l]. Note that the optimization algorithm can make W[l+2]=0 so practically a[l+1] can be removed for calculating a[l+2].

3

Thank you. So what I don’t understand is: If we use a[l+1] for calculation, how come we still say that we skip a[l+1]?

4

Hi @Doron_Modan

When we calculate the a[l+2] we regularly depending on a[l+1] only and gradient descent algorithm automatically tune w and b weights according to cost function but in the skip connection a[l+2] depending on a[l+1] also a[l] and to predict the value of this layer …the gradient descent algorithm also tune w and b and the benefit of the skip connection is if your algorithm suffer from over fitting the gradient descent set the value of the w close to zero so the value of this layer will depending on the a[l] and if your algorithm didn’t suffer from over fitting the value of this layer will depend on a[l+1] also a[l] as w and b wouldn’t be close to zero and that help the algorithm to to calculate more complex calculation and extract more high level features and the algorithm will always converge or stand and will not diverge of the so the skip connection have 2 benefits one of them is tune the over fitting of the algorithm and make sure your algorithm is always converge or stand(be fixed wouldn’t diverge)

Thanks!
Abdelrahman

5

The new technique that ResNets introduce is that there are two paths: one that skips a[l + 1] and one that doesn’t. The compute graph is no longer simply connected. Of course that affects both forward and backward propagation, as the other replies have described.

2 Likes