In the course 4, the residual networks are presented as networks that make it easy to learn the identity function from one layer to another.
Since this type of network is presented only in this course, is there a particularity that makes this type of network suitable after convolutions only or is it an idea that can be generalized to all types of networks ?
Residual networks introduce skip connections that bypass certain layers in the architecture. There are two main benefits for this type of connection. The first is that it handles better the problem of vanishing gradients, which is common to deep models. The skip connection regularizes learning by preserving the gradients early on during training, and allows them to modify weights in the regular path later during training.
Another usage for skip connections is in architectures like U-net, in which the connection directly passes input information to deeper layers. This type of information preserves features (resolution, e.g.) of the input (image) that are downsampled in the regular pass.
As far as I know skip connections are used in CNN architectures for problems in computer vision. However, remedies to the vanishing gradient problem also appear in other types of networks, but may take different forms. For example, memory cells in LSTMs - recurrent neural networks which are covered in DLS course 5.
3 Likes