Xavier Initilization formula

Course Q&A Deep Learning Specialization Improving Deep Neural Networks: Hyperparameter tun
1

Hi Sir,

@bahadir
@nramon
@eruzanski
@javier
@marcalph
@elece

what is the exact math formula for xavier initlization ? and what value should we suppose tune the hyperparameter here in this formula ?

2

Hi, @Anbu.

For Xavier uniform:

  Draws samples from a uniform distribution within `[-limit, limit]`, where
  `limit = sqrt(6 / (fan_in + fan_out))` (`fan_in` is the number of input units
  in the weight tensor and `fan_out` is the number of output units).

For Xavier normal:

  Draws samples from a truncated normal distribution centered on 0 with `stddev
  = sqrt(2 / (fan_in + fan_out))` where `fan_in` is the number of input units in
  the weight tensor and `fan_out` is the number of output units in the weight
  tensor.

There’s nothing else to fine tune, other than selecting the right initializer, of course :slight_smile: