As the course mentioned, the lower layer unit can be actived by some simple feature like lines and some simple edges. So how does the network know what kind of lower feature the layer should use to detect?
It’s a good question. The answer is that it just learns “what works” through back propagation, which is driven by the combination of the labeled training data and the cost function. We don’t know ahead of time exactly what will be the early primitive features and what the more complex features are that will be learned during training. In the Week 4 lecture titled “What are Deep ConvNets Learning?” Prof Ng shows us some really interesting research that probes individual neurons in the internal layers of a network to see which patterns most strongly “trigger” them, meaning what it is that they have been trained to detect.
The other general point to make here is that any particular neuron in any particular layer won’t necessarily learn the same thing if you run the training again from scratch. The reason is that the weights and bias values are randomly initialized, so the learning may happen differently each time. The same features will be learned from the same data, but which actual neuron recognizes a particular thing may be different.
1 Like
And why the lower layer can only learn the lower level feature? If I larger the kernel size , can the lower layer learn higher level feature? Or the lower feature is relative lower compared to features learned by deeper layers?
Have you watched the lecture I referred to in my previous reply on this thread? Prof Ng explains all this in that lecture. The point is that the way Neural Networks work is that everything happens in stages as you go through the layers. First it tries to detect simple patterns. Then as you go through subsequent layers, it assembles the more primitive patterns or features into the detection of more complex features. Just as an example to give the intuitive idea, maybe the first few layers detect edges and curves. Then in a later layer it can figure out that two edges that meet in a certain range of angles might represent the tip of a cat’s ear. Or two curves that meet at a certain angle are the corner of a cat’s eye.
But Prof Ng does a way better job of explaining all this in the “What are Deep ConvNets Learning” lecture. Please watch that or watch it again with what I said above in mind.