Course 2 Week 1 Programming Assignment Regularization

@neurogeek’s excellent answer has completely covered the issue here, but there is one caveat about any conclusions you derive from running experiments using the code that we have written here in the Regularization notebook:

Notice that the template code that they gave us sets the random seed in the forward propagation with dropout function. That means that we actually end up dropping exactly the same neurons in every iteration, so what we have implemented here is not really dropout as it was intended. The whole point of dropout is that you don’t drop the same neurons in every iteration, so that you get statistical behavior w.r.t. weakening the connections. So if you want to run experiments using the code as we wrote it here, it would be a good idea to remove the setting of the random seed so that you really see the full effect of dropout. With the fixed seed, our dropout implementation basically just subsets the network in a fixed way, which is not the real point of dropout. In fact, thinking a little more deeply about this, the “fixed dropout” is actually worse than just subsetting the network. That’s because what happens is that the fixed set of neurons we drop have weights that just never get updated, but then when we use the trained model, those weights are still there with whatever random initial values they happened to get and the training has not affected them at all. That’s because the way the dropout logic works on back propagation is that it zaps the gradients for those nodes. So it seems pretty important that we remove the setting of the random seed for any real work we intend to do with this code.

Of course this point is completely independent of the “big picture” answer that @neurogeek gave us above.