C3W1 - Lab Intro to Keras Tuner - Bonus Challenges

I tried to hypertune the dropout layer, no of layers, activation function for each layer and number of units in each layer. I used hyperband to tune the hyperparameters.
Everytime I got a different optimal results for each of these hyper parameters. I also noticied that the number of trials each time was 30. Is there a way to increase the number of trials so that all possible hyper parameter combination is evaluated? The objective is to get the same set of hyper parameters after tuning the hyper parameters. The hyper parameter space that I tried are as follows:

#Activation Function
hp_activation = hp.Choice(“activation”, values=[“selu”,“relu”,“sigmoid”,“tanh”])

#Dropout
hp_dropout = hp.Float(“dropout”, min_value=0.1, max_value=0.9, step=0.1)

#Learning Rates
hp_lr = hp.Choice(“lr”, values=[1e-5, 1e-4, 1e-3])

#Number of Layers
hp.Int(name=“dense_layer_num”, min_value=2, max_value=20)

#Number of Units in each layer
hp.Int(“units_”+str(i), min_value=16, max_value=512, step=16)

Hi @neerajkumar
Welcome to our community!
Just for my understanding, which is the range of the different optimal results for each of these hyper parameters?
About your question " Is there a way to increase the number of trials so that all possible hyper parameter combination is evaluated?" maybe you can remove the ‘stop_early’ callback.
Anyway I don’t expect improvements because early stopping is used precisely to “monitor the validation loss and stop training if it’s not improving after N epochs” (in this case 5).
Hope this can help
BR

Hi @fabioantonini,

Below are the values that I got for different runs. I ran the tuner 6 times, last two times without early stopping. Still the number of trials were 30.

Run No 1 2 3 4 5 6
Activation Function selu relu selu tanh tanh selu
Dropout Rate 0.1 0.3 0.1 0.6 0.4 0.4
No of Layers 12 2 5 14 18 6
No of Units in 1 Layer 468 128 416 336 496 336
No of Units in 2 Layer 48 144 416 208 256 144
No of Units in 3 Layer 512 480 304 480 480
No of Units in 4 Layer 128 320 160 272 304
No of Units in 5 Layer 464 496 416 112 16
No of Units in 6 Layer 224 112 272 416
No of Units in 7 Layer 192 48 240
No of Units in 8 Layer 368 272 384
No of Units in 9 Layer 416 416 32
No of Units in 10 Layer 224 64 384
No of Units in 11 Layer 288 224 384
No of Units in 12 Layer 384 400 432
No of Units in 13 Layer 240 320
No of Units in 14 Layer 304 128
No of Units in 15 Layer 448
No of Units in 16 Layer 208
No of Units in 17 Layer 144
No of Units in 18 Layer 112

Hi @fabioantonini ,

Upon some research, I found that max_trials argument is allowed in BayesianOptimization Tuner, RandomSearch Tuner and as part of Oracle argument in Sklearn tuner. But for Hyperband Tuner, which I am using, I did not find any way of specifying max_trials.

A few sites specified that the max trails in hyperband tuner depends on the max_epochs and hyperband_iterations, but I have not been able to increase the number of trials by changing either of two.

1 Like