How is the neuron units number in a layer decided?
Is it by input parameter number or the output parameter number? or just impacts the speed?
There is no magic recipe or formula. It is done from experience and by experimentation.
Do you divide training set among multiple neurons if you set more neurons than the number of output features, like 5 neurons for 3 features (perceived quality, affordability, awareness).
I’m not sure I understand your question, but you do not divide the input data between different neurons. Every neuron gets all the inputs at each layer of the neural network.
Here again, I would bet money that Professor Ng covers that point in the lectures, although I am not familiar with the MLS lectures. I know he does discuss this in DLS.
1 Like
While you are here, I would like to get to the bottom of it.
So what differentiate the different neurons, do they receive different input, in the course we have T shirt color, quality, marketing data as examples, can you use them to explain?
Thank you very much for your tutoring.
All the neurons get all the inputs. So they all get the same inputs. But they are initialized with random weights, so each neuron starts with different weight values. Then we start the training with different neurons and they each learn different things. The examples that Professor Ng gives of things that internal neurons might learn to recognize are just that: examples. We don’t know in advance what they will learn, but they will each learn different things.
1 Like
Amazing. So these neurons are like black boxes, are there ways to retrieve what actually dictate the model’s calls?
There are ways to instrument the internal structures of a neural network in order to figure out what inputs the individual neurons are reacting to, but that is a lot more advanced topic. It is described by Professor Ng in Course 4 of DLS, which is where you might choose to go after you finish MLS. If you want to get a preview, that lecture is also out on YouTube. It will talk about some things like Convolutional Nets which you have not yet heard about in MLS, but maybe the ideas will give you some useful intuitions. Or at least it will give you a preview of the sort of things you can learn by taking DLS next.
Hello Wayne @wge6729
Perhaps my answer is boring, but it is the loss function that dictates the call. The training always try to reduce the loss, so whatever neurons learn at the end is always just the best they can minimize the loss with respect to the training data. In fact, the neurons do not care if any of them were interpretable in layman terms. For example, in the slide below,
The idea that the neurons represent “affordability”, “awareness” and “perceived quality” are really an ideal case. As a MLS mentor, I could see how reasonable it is to use this example to deliver the idea of neural network, however, I would also suggest all learners not to take such “layman interpretation” for granted in an actual neural network, in fact, we cannot hope for it because, again, the training only requires minimum loss and it does not require layman interpretation.
In the same lecture, Andrew has also said the following:
One note, even though I previously described this neural network as computing affordability, awareness, and perceived quality, one of the really nice properties of a neural network is when you train it from data, you don’t need to go in to explicitly decide what other features, such as affordability and so on, that the neural network should compute instead or figure out all by itself what are the features it wants to use in this hidden layer.
You could understand “figure out all by itself what are the features it wants to use in this hidden layer” as “there is no guarantee that these features are interpretable in layman terms.”, because by “it wants”, it means how “it can minimize the training loss”.
Cheers,
Raymond
PS: @wge6729, I believe you were actually referring to the slide shared above when you asked Paul this question. Next time, if you have a slide or lab in mind, I encourage you to post a screenshot or the link to it (with time mark for lecture) so that we can use it for the discussion or we make sure we talk about the same thing.
3 Likes