Hello, given the attached picture of an RNN , how would we compute the number of parameters in the RNN cell?
HI @Zineb_Attaoui ,
I understand that you want to compute the number of parameters of a RNN, that is, the total count of parameter of a RNN.
In a recurrent neural network RNN we have to compute the number of parameters of the input layer, plus the number of parameters in the hidden layer.
In the input layer, the number of parameters comes from the size of the vocabulary, say âmâ.
In the hidden layer, the number of parameters comes from the number of units of the layer, say ânâ
Since the input is m and the hidden layer has n units, the W params of the input layer would be m * n.
Input weights: m*n
Since the network is recurrent, meaning it âcallsâ itself over and over, then the W parameters in the hidden layer will be n * n.
Recurrent weight: n*n
If we add biases, then we would have âmâ biases for the input layer and ânâ biases for the hidden layer.
To compute the total number of parameters we add (m * n) + (n * n) + m + n
1 Like
Juan has covered how to compute the total number of parameters in the RNN cell, given that you know the size of the inputs and the number of elements in the hidden state of the cell. If the question is how do we determine the number of elements in the hidden state, the answer is that it is what Prof Ng calls a âhyperparameterâ, meaning a value that you simply have to choose as the system designer. If you choose too small a value, the network may not perform very well. If you choose too large a value, it may cause the cost of training the network to be more expensive than it needs to be. So the goal is to pick a âGoldilocksâ value that is just right. If you have err, it would be better to err on the side of slightly too large a value.
2 Likes
