# Weight initialisation

**URL:** <https://community.deeplearning.ai/t/weight-initialisation/656211>\
**Category:** Improving Deep Neural Networks: Hyperparameter tun\
**Tags:** week-module-1, coursera-platform\
**Created:** [July 1, 2024, 6:11am UTC](https://community.deeplearning.ai/t/weight-initialisation/656211 "2024-07-01T06:11:37Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![computerandgyein](https://avatars.discourse-cdn.com/v4/letter/c/b4bc9f/32.png) [@computerandgyein](https://community.deeplearning.ai/u/computerandgyein)\
**Post date:** [July 1, 2024, 6:11am UTC](https://community.deeplearning.ai/t/weight-initialisation/656211/1 "2024-07-01T06:11:37Z")

</div>

In Week 1 – [Video (Weight Initialization for Deep Networks)](https://www.coursera.org/learn/deep-neural-network/lecture/RwqYe/weight-initialization-for-deep-networks),

Hello,

From this given image,

 ![image](https://global.discourse-cdn.com/dlai/original/3X/e/8/e8a9ceb9af8d54d5809ee4c60a4aec6746a141ea.jpeg)

Is there anyone can explain to me why when n gets larger, w\_i will get smaller?  
Also, the reason of taking (or assuming) the variance of w to be 1/n?

---

<div class="post-metadata">

**Author:** ![TMosh](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/tmosh/32/9321_2.png) [@TMosh](https://community.deeplearning.ai/u/TMosh)\
**Post date:** [July 1, 2024, 6:37am UTC](https://community.deeplearning.ai/t/weight-initialisation/656211/2 "2024-07-01T06:37:49Z")

</div>

Since the sigmoid(z) function has a limited output range (from 0.0 to 1.0), the gradients will approach zero when the z values are large. So when you have lots of inputs to compute w\*x and sum to get z, the weights must be learned to be small, to prevent the output getting into the range where the gradients are tiny.

---

<div class="post-metadata">

**Author:** ![computerandgyein](https://avatars.discourse-cdn.com/v4/letter/c/b4bc9f/32.png) [@computerandgyein](https://community.deeplearning.ai/u/computerandgyein)\
**Post date:** [July 1, 2024, 7:29am UTC](https://community.deeplearning.ai/t/weight-initialisation/656211/3 "2024-07-01T07:29:08Z")

</div>

Thank you @TMosh,

Ohhhh you’re right, that makes sense.

---

<div class="post-metadata">

**Author:** ![tarunsaxena1000](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/tarunsaxena1000/32/84439_2.png) [@tarunsaxena1000](https://community.deeplearning.ai/u/tarunsaxena1000)\
**Post date:** [July 1, 2024, 3:46pm UTC](https://community.deeplearning.ai/t/weight-initialisation/656211/4 "2024-07-01T15:46:41Z")

</div>

> [@computerandgyein](#):
>
> the reason of taking (or assuming) the variance of w to be 1/n

- Consider a single neuron with n input connections. The output of this neuron is typically a weighted sum of the inputs plus a bias term.
- If the inputs have zero mean and unit variance, and if the weights are initialized with zero mean and variance σ^2, the variance of the weighted sum (output of the neuron before applying the activation function) is nσ^2.
- To maintain a unit variance for the output, we set n x σ^2=1. Thus, σ^2=1/n.

This helps in preventing the gradients from either vanishing or exploding as they propagate backward through the network.

_This is just my intuition as I am also in the learning phase, for details we need to read the research paper._