How to sample hyperparameters? (DOUBT)

[rmoret]

Hi! I’ve just completed the Week 3 Quiz of Course 2 and I have a doubt about the sampling technique that we could use to select our net hyperparameters.

If we would want to sample a hyperparameter from 0.9 to 0.99 why shouldn’t we use a formula like this:

[Edited to remove the formulas]

And instead use this:

[Edited to remove the formulas]

(where r is a uniform random number between 0 and 1)

I’ve calculated some examples of both and although they are not identical they output quite similar results so I don’t understand why one formula is better than the other.

Some results I’ve calculated:

* r = 0:     0.9     0.9
* r = 0.1:   0.909   0.92
* r = 0.5:   0.945   0.968
* r = 0.7:   0.963   0.98
* r = 1:     0.99    0.99

Any help to clarify this? Thanks!

[nramon]

Hi, @rmoret.

The second formula is better because you don’t want to sample uniformly in that interval. Since values of Beta close to 1 have a greater impact on the outcome, you want to sample more densely in that region (your goal is to better explore the space of possible outcomes).

To clearly see the difference, you need to generate a lot more samples. For example, 10000:

[Edited to remove the formulas]

Let me know if that makes things clearer

[rmoret]

Thanks, @nramon, much clearer now!

The plots help a lot to understand it, thanks for taking the time to make them.

[nramon]

My pleasure!

Would you mind editing your post and removing the images for the formulas so that we don’t give away the answer? I’ll do the same.

Enjoy the rest of the specialization!

[rmoret]

Yes, for sure!

Thanks!