In mini-batch gradient decent we process a mini-batch, then the secon, and so on in every epoch. Say there are 1000 epochs, why can’t we run 1 mini-batch for 1000 epochs, then the second batch for 1000 epochs, and so on? This will reduce the need to swap the data in and out of GPU memory. The algo…

You need to go through all of your data several times to learn from it. If you use only a minibatch per epoch you might be going one once through your data (or may a few times), but the model needs to go through it many times to effectively learn from it!

So eventually the model does go through all data. Steps: Take mini batch 1 - train for 1000 epochs Take mini batch 2 - train for 1000 epochs and so on. So when the algorithm finishes, the network has seen all data.

The effect of doing it that way will be much different and I would bet will not work very well. The order matters. The point is that the minibatches are much much smaller than the complete data set. E.g. typically 32 entries or fewer. That means that they may not be statistically representative of t…

Why do we process all minibatches in an epoch?

Course Q&A Deep Learning Specialization Improving Deep Neural Networks: Hyperparameter tun

balaji.ambresh October 14, 2025, 4:16pm 5

Tiny detail:

We don’t have to process all mini batches of underlying data in every epoch (please see this topic)

See this link for more onsteps_per_epoch parameter.

Topic		Replies	Views
Possibility of loosing Effectiveness of large datasets in Mini Batch Improving Deep Neural Networks: Hyperparameter tun week-module-2 , coursera-platform	6	48	May 11, 2025
Gradient steps in Mini batch vs batch Improving Deep Neural Networks: Hyperparameter tun coursera-platform	4	867	May 18, 2021
Batch vs MiniBatch Improving Deep Neural Networks: Hyperparameter tun coursera-platform	1	545	November 1, 2021
Why is Mini-batch Gradient Descent more efficient? Improving Deep Neural Networks: Hyperparameter tun coursera-platform	4	602	December 24, 2022
Mini-batch understanding Improving Deep Neural Networks: Hyperparameter tun coursera-platform	8	764	March 7, 2023

Why do we process all minibatches in an epoch?

Related topics