# Exponentially Weighted Averages: Initial Values

**URL:** <https://community.deeplearning.ai/t/exponentially-weighted-averages-initial-values/16457>\
**Category:** Improving Deep Neural Networks: Hyperparameter tun\
**Tags:** coursera-platform\
**Created:** [June 22, 2021, 5:06pm UTC](https://community.deeplearning.ai/t/exponentially-weighted-averages-initial-values/16457 "2021-06-22T17:06:03Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![xuanghdu0](https://avatars.discourse-cdn.com/v4/letter/x/71c47a/32.png) [@xuanghdu0](https://community.deeplearning.ai/u/xuanghdu0)\
**Post date:** [June 22, 2021, 5:06pm UTC](https://community.deeplearning.ai/t/exponentially-weighted-averages-initial-values/16457/1 "2021-06-22T17:06:03Z")

</div>

Hi! I am wondering for exponentially weighted averages, why cannot we initialize v\_0 = \theta\_1 instead of 0 so that the curve can pass through the first point even without bias correction? Thanks for any help!

---

<div class="post-metadata">

**Author:** ![nramon](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/nramon/32/129097_2.png) [@nramon](https://community.deeplearning.ai/u/nramon)\
**Post date:** [June 23, 2021, 3:58pm UTC](https://community.deeplearning.ai/t/exponentially-weighted-averages-initial-values/16457/2 "2021-06-23T15:58:18Z")

</div>

Hi, @xuanghdu0.

One is equivalent to having seen an infinite sequence of 0s, the other is equivalent to having seen an infinite sequence of \theta\_1.

Doesn’t explicitly accounting for the bias introduced intuitively make more sense? (I could be wrong.)

````nohighlight
  All exponential moving averages initialized with Tensors are initialized to 0,
  and therefore are biased to 0. Variables initialized to 0 and used as EMAs are
  similarly biased. This function creates the debias updated amount according to
  a scale factor, as in (Kingma et al., 2015).
  To demonstrate the bias the results from 0-initialization, take an EMA that
  was initialized to `0` with decay `b`. After `t` timesteps of seeing the
  constant `c`, the variable have the following value:
  ```
    EMA = 0*b^(t) + c*(1 - b)*b^(t-1) + c*(1 - b)*b^(t-2) + ...
        = c*(1 - b^t)
  ```
  To have the true value `c`, we would divide by the scale factor `1 - b^t`.

````

([source](https://github.com/tensorflow/tensorflow/blob/a4dfb8d1a71385bd6d122e4f27f86dcebb96712d/tensorflow/python/training/moving_averages.py#L198-L212))

Take a look at section 3 of the cited [paper](https://arxiv.org/pdf/1412.6980.pdf) if you’re interested in the complete derivation.