Keras-tuner hyperparameter tuning

AI Discussions
1

I’m using keras-tuner for deciding the optimum learning rate and optimizer for my use case. Following are the details:-

image
image1026×416 38.6 KB

image
image1571×390 33.8 KB

I run 5 trials with 3 epochs each. While training I noticed something unusual. Theoretically, in each trial a new pair of hyperparameters are picked from the search space that I defined and model is trained from scratch. But notice the anomaly in the 2nd trial below:

image
image1203×474 35.2 KB

Training loss during the 1st epoch of 1st trial was around 0.96 which declined to 0.6 at the end of 1st trial. Now, the training loss shall have similar values at the start of 2nd trial since the model is trained from scratch with a new pair of hyperparameters, but it seems like training has resumed from the end of 1st trial (see the loss value at the beginning of 2nd trial). This is counter-intuitive. I read the official documentation which too states that During the search, the model-building function is called with different hyperparameter values in different trial. In each trial, the tuner would generate a new set of hyperparameter values to build the model. The model is then fit and evaluated.
What then is the problem with in my case? Why does it seem like with each trial, instead of training the model from scratch, it simply continues from the previous trial?

2

Hi @Harshit1097 ,
Using only two epochs to decide the direction of the loss function is not a good approach. In some cases, the model may show instability during training and it is a good idea we can to remove this instability. For short, if you are just doing an initial training to see where the loss function goes, try using something like 10 epochs or so to ensure you have collected a piece of minimum information about the loss function behavior.

1 Like
3

I also have a question about the hyperparameter tuning with keras-tuner.
I want to resume the tuning process after the computer has been switched off and on again for a short time. For this purpose “overwrite=False” during search should be set. In fact, it continues with the next trial, but there is an error and the training ends with an error message. Have you also observed this behavior and if so, maybe a solution for it?

4

Hi @Volker_Drewes , the best way to help is that you to post the error message here. Anyway, you can set to save your best model during each epoch and you can load this model (or weights) during the resume process. Another approach is to use a monitoring tool like the weights & biases that will automate this process and do all the necessary tasks for you.
Keep learning!