BatchNorm at Test set (choose mean and var)

Course Q&A Deep Learning Specialization Improving Deep Neural Networks: Hyperparameter tun
1

I would like to ask about the mu and var using for the test set.

Well, we use exponentially weighted average to predict mu and var ( base on the minibatches)

running_mean = momentum * running_mean + (1 - momentum) * sample_mean
running_var = momentum * running_var + (1 - momentum) * sample_var

But, like this, we’ll have lots of pairs (mean,var) depends on which minibatches we’re currently on .

So, with the Test set, we just use the last pairs, the pairs corresponding the last mini-batches ? Or which pairs we should use ?

Thank you, I hope you could help me with this!

2

Hi, @jacknguyen101.

You use the latest values you have computed. If you look closely at the formulas, you’ll see you’re updating running_mean and running_var over time. If, for some reason, you wanted to keep all the intermediate values, you’d have to explicitly save them somewhere else (as @ajaykumar3456 did when he ran his own experiment).

You may find this post useful too.

Let me know if that helped :slight_smile:

1 Like
3

Thank you so much for your answers!

1 Like