In the lesson “Improving your model’s performance” from the course “Structuring Machine Learning Projects – DeepLearning.AI”, I have the following questions:
At which stage should the process of hyperparameter tuning take place, including learning rate, momentum, minibatch size, and number of hidden units? (Figture below)
In the previous course, the instructor mentioned that the algorithm was among the least important hyperparameters. Why does he now say in this lesson that it should be considered second in importance? (Figture below)
During training, before the final testing stage. First, you try to improve the model’s performance during the training stage, i.e., you measure the model’s performance and change hyperparameters after every training run. During training, you test the model with the training and dev datasets!
Because changing the algorithm is not an easy thing to do, it is the foundation of your model, and if you change that, then you need to rethink all other hyperparameters! I believe he says this in this context!
Regarding the first answer:
We know that during training we evaluate the bias, and with the development set we evaluate the variance.
So, if the bias is high, we need to reduce it — which can be done by testing a more complex model or increasing the amount of data.
On the other hand, if the problem is high variance, we can apply regularization or adjust the architecture of the model.
But at what stage does hyperparameter tuning come into play in this process?
Regarding the second answer:
We also know that among the hyperparameters, some are more important than others — for example, learning rate, momentum, minibatch size, and number of hidden units.
Meanwhile, the training algorithm is usually considered less relevant.
So why does Ng mention that, as a second measure, one could change the algorithm?
“I want to understand how all of this fits together; please keep this logic in your response.”
At which stage should hyperparameter tuning take place, including learning rate, momentum, minibatch size, and number of hidden units?
Hyperparameter tuning is part of the training cycle and comes before the final testing stage. The process is iterative: train the model, evaluate performance on the training and development sets, adjust hyperparameters, and retrain. Which hyperparameters you focus on depends on whether the main issue is high bias or high variance. For example, if bias is high, you may increase model complexity by tuning the number of hidden units or depth. If variance is high, you may adjust regularization strength, dropout, or batch size. Thus, tuning occurs throughout training as a means of systematically reducing bias or variance before final evaluation on the test set.
Why does Ng place the algorithm second in importance, even though earlier he said it was less relevant?
In practice, hyperparameters such as learning rate or batch size usually have a stronger effect on model performance than switching optimization algorithms. However, algorithm choice is ranked second in importance because it forms the foundation of the model. If the algorithm family itself is a poor match for the task—for example, using linear regression for image classification—no amount of hyperparameter tuning will fix the problem. Changing the algorithm is therefore a bigger, structural decision: it resets what hyperparameters matter and requires rethinking the training process. This explains why Ng emphasizes its importance, while still acknowledging that, once an algorithm is chosen, hyperparameter tuning often has more day-to-day impact.
