Backprop: no context around the 'step' notation

Course Q&A Natural Language Processing NLP with Probabilistic Models
1

Hi,

Looking at the assignment exercise for backpropagation, I also went back to the lecture, and I’m seeing this:

image
image936×432 47.8 KB

However, I don’t think the ‘step’ function was ever explained. I’m not sure how to go about implementing the back prop function.

Thank you for any input.

Edit: My updated understanding is that the ‘step’ part is was is implemented in the assignment when going from variable z1 to l1.

2

Hi @Luc2357,
The step function is needed for the backward propagation through the ReLU non-linearity. For every positive element of Z_1 it should output one, and for other elements it should output zero.

This ensures that only the positive elements​ contribute to the gradient flow.