Implementation keep_prob in dropout

[kampamocha]

Hi @jijo,

Given the random nature of the process, that could happen. You have no guarantee that the droped neurons would be exactly 20%, it could be more or less depending on randomness.

There have been some discussions about it in this forum, so I encourage you to take a look at those. Some posts that could be helpful are below, but I’m sure there’s more.

Week 1 - Doubt in Dropout Regularization lecture video

No need to apologize! The problem of symmetry is something different. Dividing by keep_prob is compensating for this: [image] (source) From what I understood, this is the formal justification, but an example may be clearer: >>> m = 1000 >>> keep_prob=0.5 >>> r = np.random.binomial(1, keep_prob, m) >>> y = np.random.normal(1, 0.1, m) >>> np.mean(y) 1.0004794606768865 >>> np.mean(r * y) 0.5123941559332359 >>> np.mean(r * y / keep_prob) 1.0247883118664718 Let me know if something doesn’t make …

Backward_Propagation_With_Dropout

Hi, While using inverted dropout for regularization, during every iteration of gradient descent we are already dropping out and scaling activations during the forward pass. Why is it needed to explicitly set the switched off dAs to 0 and then scale the dAs in the backward pass? Shouldn’t it automatically happen as a result of the dropouts in the forward pass? My logic: If we implement inverted dropout on AL then shouldn’t dAL already have the corresponding dropped elements zero’d and the rema…