Increasing the parameter keep_prob from (say) 0.5 to 0.6

Course Q&A Deep Learning Specialization Improving Deep Neural Networks: Hyperparameter tun
1

Why increasing the parameter keep_prob from (say) 0.5 to 0.6 will likely cause the neural network to end up with a lower training set error?

Thanks in advance

1 Like
2

Using enable_op_determinism in your notebook, what are your results?

1 Like
3

Because a higher keep_prob value means you are doing less regularization. Meaning less of a dropout effect. If you set keep_prob = 1, then you are just using the model as is and not getting any regularization effect. But presumably the reason you are experimenting with drop out in the first place is that you have an overfitting problem, so you need a better balance between training set accuracy and dev or test set accuracy.

4

Let’s look at it this way, to prevent overfitting when training neural network, dropout (regularization) tends to take the hidden node of neural network towards zero if you’re using L2 regularization. So keep_prob is the probability of keeping the node when training the network (i.e don’t apply regularization). So the closer you set keep_prob to 1 the more chances of not droping the node (i.e no making it towards 0 for instance.)