I’m debugging my code for gradient_check_n and I’m looking at calculations at each step.
What I’ve found is a difference between grad and ```gradapprox `` values - but not for all records. For some this is almost perfectly aligned but for some there is a difference of scale 2. See below:
What might be the reason for such a behaviour?
Each row of gradapprox is being calculated as:
gradapprox[i] = (J_plus[i] - J_minus[i])/(2*epsilon)
I know there should be an error but the expected value of the check is wrong. What I get is:
Or is it an error in the difference calculations?
I’ve never looked at the detailed elements of the grad and gradapprox. So it looks like you have not corrected the intentional mistakes they put in backprop, but that is fine. The goal is to get one of the two correct answers (the one with the error or the one with the errors fixed).
So this says that there is something else wrong with your calculations. Have another close look at the earlier code before the gradapprox[i] line you show, which looks correct to me.
Just out of curiosity, I added similar instrumentation to my code and here’s what I get:
32: gradapprox[i] = [0.], grad[i] = [0.]
33: gradapprox[i] = [0.], grad[i] = [0.]
34: gradapprox[i] = [0.], grad[i] = [0.]
35: gradapprox[i] = [0.9158017], grad[i] = [1.8316033]
36: gradapprox[i] = [0.02451541], grad[i] = [0.04903095]
37: gradapprox[i] = [-0.10797954], grad[i] = [-0.21595908]
38: gradapprox[i] = [0.90281879], grad[i] = [1.80563782]
39: gradapprox[i] = [0.], grad[i] = [0.]
40: gradapprox[i] = [0.], grad[i] = [0.]
41: gradapprox[i] = [0.], grad[i] = [0.]
42: gradapprox[i] = [0.19763344], grad[i] = [0.19763343]
43: gradapprox[i] = [0.], grad[i] = [0.]
44: gradapprox[i] = [0.], grad[i] = [0.]
45: gradapprox[i] = [2.24404227], grad[i] = [2.24404238]
46: gradapprox[i] = [0.21225742], grad[i] = [0.21225753]
numerator = 2.3225406048733928
denominator = 8.146602433873603
There is a mistake in the backward propagation! difference = 0.2850931567761623
Interestingly it looks like my values agree with yours to the resolution you show for the individual gradapprox[i] values. It is the denominator that is wrong in your case. There aren’t very many moving parts there. 
1 Like
Thank you,
indeed, there was a typo in denominator code. 
