Thank you very much!
May i ask another question:
So in this example: image|690x362
Lets say w and b are initialized with zeros and lets assume that the very first Pixel of an 64x64x3 image is 250 and the second is 100. So the first pixel (x1/x2) out of 12288 is 250/100. Lets assume alpha is 0,05.
Lets say after the forward propagation “a” or “y-hat” = 0,80 and (the underlying image) y=1.
So by performing backpropagation I want the model to change the parameters in order to get y-hat closer to 1 than what it is right now with 0,80.
So dz = y-hat - y = 0,80 - 1 = -0,20
dw1 = x1*(a-y) = 250*-0,20= - 50
dw2 = x2*(a-y) = 100*-0,20 = -20
w1 was zero and is now: w1 = 0 - 0,1*-0,20250 = 5
w2 was zero and is now: w2 = 0 - 0,1-0,20*100 = 2
Next forward propagation is z = w1x+b
z2 for x1 = 2505+b = 1000+b
z2 for x2 = 100*2+b = 200+b
→ which leads to a greater z than before which leads to a greater sigmoid of z, right? This makes sense because we want w to amplify x1 and x2 because they already perform ok?
but what if the modell performes poorly. lets assume y-hat is 0,2 so
dz is 0,2-1 = -0,8
dw1 = x1*(a-y) = 250*-0,8= - 200
dw2 = x2*(a-y) = 100*-0,81 = -80
w1 was zero and is now: w1 = 0 - 0,1*-200 = 20
w2 was zero and is now: w2 = 0 - 0,1*-80 = 8
z2 for x1= 25020+b = 5000+b
z2 for x2 =1008+b= 800+b
which amplifies the Modell even more. What am i missing?
Thank you very much! I hope what i wrote here does make sense :).