Here, I’ve just used another way to calculate \frac{\partial{L}}{\partial{w}} and acquire the same result by some transformations and direct derivation:
I know this is not practical way but I just want to strengthen my understanding by looking for a pure substitution method. Please correct me if I was doing something wrong
\eqalign{
L &= -[y * ln(\frac{1}{1 + e^{-(wx+b)}}) + (1 - y) * ln(1 - \frac{1}{1 + e^{-(wx+b)}})] \\
&= -[y * ln(\frac{1}{1 + e^{-(wx+b)}}) + (1 - y) * ln(\frac{e^{-(wx+b)}}{1 + e^{-(wx+b)}})] \\
}
Now, I used the fact that ln(\frac{1}{1 + e^{-z}}) = -ln(1 + e^{-z}) and ln(\frac{e^{-z}}{1 + e^{-z}}) = -z - ln(1 + e^{-z}), where z = wx+b:
\eqalign{
L &= -[-y * ln(1 + e^{-(wx+b)}) + (1 - y) * (-(wx+b) - ln(1 + e^{-(wx+b)}))] \\
&= -[-y * ln(1 + e^{-(wx+b)}) - (1 - y) * ln(1 + e^{-(wx+b)}) - (1 - y) * (wx+b)] \\
&= ln(1 + e^{-(wx+b)}) + (1 - y) * wx + (1 - y) * b] \\
}
Now, find \frac{\partial{L}}{\partial{w}}:
\eqalign{
\frac{\partial{L}}{\partial{w}} &= \frac{-x * e^{-(wx+b)}}{1 + e^{-(wx+b)}} + (1 - y) * x] \\
&= \frac{-x * e^{-(wx+b)} + (1 - y) * x * (1 + e^{-(wx+b)})}{1 + e^{-(wx+b)}} \\
&= \frac{-x * e^{-(wx+b)} + (1 - y) * x + (1 - y) * x * e^{-(wx+b)}}{1 + e^{-(wx+b)}} \\
&= \frac{-xy * e^{-(wx+b)} + x - xy}{1 + e^{-(wx+b)}} \\
&= \frac{-xy * (1 + e^{-(wx+b)}) + x}{1 + e^{-(wx+b)}} \\
&= -xy + \frac{x}{1 + e^{-(wx+b)}} \\
&= (\frac{1}{1 + e^{-(wx+b)}} - y) * x \\
&= (sigmoid(z) - y) * x = (f - y) * x
}