I have completed the MLS and begun week one of the DLS and now have a question going back to this module from week one of the Advanced Learning Algorithms course.
Please tell me if this is correct:
When one feeds a feature vector X, as input to a dense hidden layer with the activation function ReLu, each node will fit a ReLu function through every feature in vector X and provide a prediction. Each of these predictions are organized into a new vector a[1], which is used as input for the next hidden layer, and so on.
If this is correct wouldn’t the output vector a[1], be a vector of length equal to the number of nodes in hidden layer one, and each element in vector a[1] be the same number?
In a dense hidden layer, every feature from vector X, was used in every node therefore every node will produce the same output value in vector a[1]. Specifically elements a[1]_1, a[2]_2 … a[j] are all the same number.
That just feels like it can’t be correct. Can someone help me understand better? Thanks for the help!
No, that’s not what a hidden layer does. Predictions are only formed at the output layer. Every unit has an activation value, but only in the output layer is that a prediction.
Each pair of adjacent layers in a neural network are connected by a weight matrix.
The size of the weight matrix is {outputs by inputs}, where outputs is the number of units in the next layer, and inputs is the number of units in the previous layer.
The number of units in each hidden layer is independent. You select that as part of designing the model.
The number of units in the input and output layers are determined by the number of features (the input), and the number of outputs (i.e. the number of labels).
Suppose each hidden layer in this NN diagram is a dense hidden layer with the same activation function, ReLu. Also, suppose that vector X, has 100 features.
Please tell me if these statements are correct:
Vector a[1] is the output of hidden layer 1 and will have length 25. Correct.
Vector a[2] is the output of hidden layer 2 and will have length 15. Correct.
Elements a[1]_1, a[1]_2, … a[1]_25 are all the same value. Incorrect.
Elements a[2]_1, a[2]_2, … a[2]_15 are all the same value. Incorrect.
Thanks for your patience and willingness to help me out.
The weight (and bias) values are learned which minimize the cost at the output layer. The method is called “backpropagation of errors”. The math involves a lot of calculus, which most ML courses assume is already proven and taken as fact.
This diagram doesn’t show the bias value that is included with each unit.
Each unit computes the sum of the product of its weight and the input values, plus a bias, with some activation g(…) applied. In vector algebra, it gives you the whole result as a vector for each example.
For the hidden layer A1, it’s A1 = g(W1 * X + b).
So what you actually get at A1 is a matrix of size (m x 3), since ‘m’ is the number of examples (rows) in the training set X.
At the output layer, there will be an additional process to compute predictions. For example, if you’re doing classification, then A2 will be turned into logical true/false values by applying some threshold operation >= 0.5. This happens because sigmoid() has a range of 0 (False) to 1 (True), and 0.5 is the boundary right in the middle.
@cwebster, one more key point is that the weights and biases need to be randomly initialized before training. Otherwise, if they are all the same and the inputs are the same, then each node in a layer will learn the same thing. It sounds like this may be the missing piece that was puzzling you.
