Determine size of n_a


I was working through week 1โ€™s programming exercises and I realized I wasnโ€™t sure how n_a is determined. The size of activations for RNNs is defined as (๐‘›๐‘Ž,๐‘š,๐‘‡๐‘ฅ) but how is na determined?

The size of the activations is a โ€œhyperparameterโ€, meaning that it is simply a choice you need to make as the system designer. You need to choose a value that captures the complexity of the โ€œstateโ€ that you need to track in the nodes of your RNN. The way such choices are made is by experience and intuition. Then you check whether your choices are good by the performance of your resulting trained model. If you choose too small a value, the model may not perform very well (โ€œunderfittingโ€). If you set it too large, then it is more costly to train your network.

1 Like