Batch Norm at test time - why exponentially weighted average instead of simple average?

Course Q&A Deep Learning Specialization Improving Deep Neural Networks: Hyperparameter tun
1

I understand the momentum value brought by the exponentially weighted average when sequentially adding pieces of information and giving more weight to the previously processed information as a whole versus to the last piece of information being considered.

But in the case of selecting a mean and variance to normalize the test examples the random sequence of batches used during training does not seem to be relevant. In fact in this case the exponentially weighted average is penalizing for no reason the batches that randomly resulted to be the last ones.

Could a regular arithmetic average of the means and variances of all mini-batches be a better hyper-parameter to be used at test time (and at production time)?

2

Training and test sets should come from the same distribution. You’ll learn more about this in the 3rd course of the specialization.

Model parameters should be learnt at train time based on your choice of hyperparameters.
Test dataset is meant for evaluating model performance. You cannot change model configuration post training and just before evaluating it.