# Numerical Approximation

**URL:** <https://community.deeplearning.ai/t/numerical-approximation/694367>\
**Category:** Improving Deep Neural Networks: Hyperparameter tun\
**Tags:** week-module-1, coursera-platform\
**Created:** [September 18, 2024, 2:37pm UTC](https://community.deeplearning.ai/t/numerical-approximation/694367 "2024-09-18T14:37:23Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![abdou\_brk](https://avatars.discourse-cdn.com/v4/letter/a/bbce88/32.png) [@abdou\_brk](https://community.deeplearning.ai/u/abdou_brk)\
**Post date:** [September 18, 2024, 2:37pm UTC](https://community.deeplearning.ai/t/numerical-approximation/694367/1 "2024-09-18T14:37:23Z")

</div>

Hello,  
I have a question about the numerical approximation and the examples prof andrew gave. I don’t understand why we take J(w+epsilon)-J(w-epsilon) rather than J(w+epsilon)-J(w) because normally the derivative is defined mathematically as the limit of J(w+epsilon)-J(w)/epsilon (epsilon–\>0). I think it makes more sense than J(w+epsilon)-J(w-epsilon)/2\*epsilon,didn’t it?

---

<div class="post-metadata">

**Author:** ![paulinpaloalto](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/paulinpaloalto/32/187_2.png) [@paulinpaloalto](https://community.deeplearning.ai/u/paulinpaloalto)\
**Post date:** [September 18, 2024, 2:46pm UTC](https://community.deeplearning.ai/t/numerical-approximation/694367/2 "2024-09-18T14:46:19Z")

</div>

It’s been a while since I watched these lectures, but I’m pretty sure Andrew comments on that in the lectures. It’s the difference between a “one sided” difference and a “two sided” difference. When we are doing “real” calculus using \mathbb{R} and are taking limits as \Delta x \rightarrow 0, we don’t really have to worry because we have infinite resolution. But when we’re operating in the limited world of 64 bit floating point and have literally only 2^{64} numbers that we can represent between -\infty and +\infty, then we have to worry about this. It just turns out that when you are estimating derivatives with finite differences, the two sided differences give you better convergence behavior. This turns out to matter a lot, because packages like TF and PyTorch use numerical approximation of gradients extensively, so this behavior has been studied carefully.

[Here’s a thread](https://community.deeplearning.ai/t/derivatives-lab-question/477894) with a relevant question from a different course, but which shows some experiments that demonstrate the behavior. You can probably also find articles on Wolfram or MathWorks by googling “two-sided difference”.

---

<div class="post-metadata">

**Author:** ![abdou\_brk](https://avatars.discourse-cdn.com/v4/letter/a/bbce88/32.png) [@abdou\_brk](https://community.deeplearning.ai/u/abdou_brk)\
**Post date:** [September 18, 2024, 2:49pm UTC](https://community.deeplearning.ai/t/numerical-approximation/694367/3 "2024-09-18T14:49:46Z")

</div>

Got it, thank you !

---

<div class="post-metadata">

**Author:** ![paulinpaloalto](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/paulinpaloalto/32/187_2.png) [@paulinpaloalto](https://community.deeplearning.ai/u/paulinpaloalto)\
**Post date:** [September 18, 2024, 2:59pm UTC](https://community.deeplearning.ai/t/numerical-approximation/694367/4 "2024-09-18T14:59:57Z")

</div>

Yes, Prof Ng does discuss this point starting about about [1:30 in the first lecture on this in DLS C2 W1.](https://www.coursera.org/learn/deep-neural-network/lecture/XzSSa/numerical-approximation-of-gradients)