Hello, I am working to complete the 2 layer network for the last assignment in week 4 however I am getting stumped where I need to compute the derivative of the relu activation. I linked my notebook below where I currently am.
My thoughts on computing the derivative of the relu activation function isnt adding up and I wanted to get some clarification…If we have a 2-layer neural network and layer 1’s activation is relu and layer 2 (output) is using sigmoid then if i am thinking about this correctly, wouldn’t relu’s derivative function set all the value of A2 to 0? (since sigmoid will be less than 1) That isnt what we want right? Is there a different variable I should be passing into the relu_backwards function?
Sorry for the loaded question, trying to understand this the best I can. I tried searching on my own, came up short.
You have the wrong idea about ReLU: the inflection point is at 0, not 1, right? So the derivative is 0 for z < 0 and 1 for z \geq 0, right? And what does sigmoid have to do with the derivative of ReLU? The inputs to ReLU are not the outputs of sigmoid, right? They are the linear outputs of hidden layer in question, which can be any z with -\infty < z < \infty.
BTW publishing your notebooks on github is a violation of the Honor Code unless you have a private github account. If it’s not private, you are sharing your solutions with the entire known Universe, which is why that is not supposed to happen.
Also note that they already wrote the relu_backward function for you. All you have to do is understand the definition of the function so that you know what to pass it. You can examine the code by clicking “File → Open” and having a look around. You can deduce the file to open by examining the import cell at the beginning of the notebook.
But thinking about this one level deeper, you don’t even call relu_backward directly in this exercise, right? You only call linear_activation_backward, which they have also provided for you.
So it seems like you are approaching this at the wrong level somehow. Perhaps it’s time to take a few deep cleansing breaths and then read the instructions in the notebook again from the beginning with a calm mind.