One thing discussed in a couple videos was that in deep neural networks, later layers might effectively identify “features” in images; e.g. certain lines that match parts of numbers in image recognition. On the other hand, I’ve seen videos on NNs that said that they don’t do this in practice. What I saw suggested that hidden layer results tend to be much more high variance and black box, very hard to predict for any kind of input.
Has more analysis been done on this? Are there any examples of someone dissecting a neural network to see which inputs produce high/low values for each hidden neuron?
It’s a very interesting question, of course! Yes, there has been work instrumenting the internal layers of networks to get a better understanding of what is happening. The approach is exactly as you suggest: instrumenting neurons in hidden layers and then feeding inputs through the net in prediction mode and noting which inputs trigger the largest activation values. Prof Ng has a great lecture on this that is specific to ConvNets in Week 4 of Course 4 of DLS. The title of the lecture is “What are Deep ConvNets Learning?” Even though it is specific to Convnets, I think you’d still be able to understand the points he’s making even without having taken Course 4. Here’s a video of the lecture that Prof Ng has posted on YouTube. Please have a look!
Much appreciated, that is exactly the kind of analysis I was wondering about.