First lesson, 4 questions about NN

Course Q&A Machine Learning Specialization Advanced Learning Algorithms
1
  1. Do neurons have to do logistic regression and not linear regression?
  2. Can a single neuron contain a NN itself? (a sort of nested NN)
  3. If the first hidden layer receives all inputs how do the different neurons decide which input to use? How come not all neurons in a layer perform the same model with the same inputs?
  4. j is the feature number of the features between 1 and n. Why does j also represent the number of neurons and how do we avoid confusion?
2

Hi @sdabach

No, it depends on whether an activation function is used. Without any activation, a neuron performs linear regression, computing a weighted sum of inputs. If you apply a sigmoid activation, the neuron behaves like logistic regression (for binary classification). In hidden layers, we use nonlinear activations like ReLU or tanh to give the network the ability to learn nonlinear patterns.

No, a single neuron is a simple unit that computes a dot product followed by an activation.

All neurons in a layer receive the same input vector but each one has its own set of weights and bias. During training, the backpropagation algorithm adjusts these weights differently for each neuron based on the error gradients they contribute to. This means neurons learn to focus on different combinations of inputs or detect different features.

In many explanations, j is reused both as the index for input features and for neurons in a layer, but they are context-dependent. When we say x_j, j indexes input features; when we say z_j or a_j, j indexes neurons.

Hope it helps! Feel free to ask if you need further assistance.

3

Helps and clarifies
Thanks

4

You’re welcome! happy to help :raised_hands:

5

It might also be worth going one level deeper on that point. The reason this happens is that we must start back propagation by randomly initializing all the weight values. That is a technique called “Symmetry Breaking”. If you didn’t do that and started all the neurons with the same weights, then they would all learn the same thing and your network would be basically useless because each layer has the equivalent of only one unique neuron. Here’s a thread which discusses Symmetry Breaking at bit more.

6

I was thinking about this but didn’t know how to phrase it into a question.
Randomizing the weights makes sense…I didn’t get to the training week yet.

Also regarding this:

j is the feature number of the features between 1 and n. Why does j also represent the number of neurons and how do we avoid confusion?"
.

If we call the input layer a_0 then we avoid the confusion in j since the inputs and the neurons in a layer have the same dimensions
Thanks

7

It’s entirely possible that I’m just missing your point in that statement, but note that is not true in general, right? There is no reason why the number of neurons in one of the hidden layers of the network needs to match the number of inputs to that layer. The typical pattern in a Fully Connected Feed Forward network is that the number of output neurons in a hidden layer is usually less than the number of inputs.

8

Got it
Thanks