# Interpretation of this learning rate finder plot

I have the following plot relating to learning rate finder results (following the principles of Smith (2015)), made according to this code example and found the loss drop section to be very narrow, what does that mean? The optimal learning rate found is 1.5655376e-05 and the batch size wrt code that was executed is 512.

Now I made the same plot using Plotly to see it better, as follows, I zoomed in on the section of interest, and I verified that the optimal learning rate is not even in the section with an abrupt drop, it is located much earlier (trying to understand why the optimal learning rate chosen is far from the abrupt drop).

I canâ€™t really see your images clearly, and youâ€™re uploaded them to a 3rd party site that my browser is blocking.

Also, please update your plot format to include markers for each data point.

@TMosh Hi, ok, I made the adjustments you requested by updating my post. Thanks!

Thanks.

First I doubt that testing the learning rate with such fine increments is really necessary. Itâ€™s not that critical to have an exact optimum value. Just getting close is good enough.

Typically youâ€™d use an approximate log range of values using ratios of 1:3:10.
So maybe use 0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1.0. If the data set is normalized, that should cover all the likely values.

Regarding your plots and why the best learning rate doesnâ€™t line up with the step function in the cost plot, I suspect thereâ€™s an error in your code.

Note that normalizing the dataset is a critical first step. Doing this will make it a lot easier to find the best learning rate.

And remember, youâ€™re looking for â€śgood enoughâ€ť. There should be a big range of learning rate values that give just about identical performance.

What do you mean exactly by testing the LR with such fine increments? Would it be possible for you to reference what you are saying in relation to the code I provided?

Do you mean to adjust these log scales in relation to simply viewing the values on the plot? And yes, I am normalizing my data set.

Are you referring to a possible problem in the code only in relation to the red line that denotes the optimal learning rate?

Ok, Iâ€™m normalizing the data set. Yes, being good enough would be enough, of course.

Thanks!

Your plot shows that youâ€™re testing with learning rates that are spaced at approximately 0.00000003 units.

Interval: 15.61e-6 - 15.58e-6 = 3.0e-8

I think that is not necessary or useful.

Yes. Start by only using these learning rates:
0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1.0

Once you identify the interval where the minimum lies, you might then test again using a few points within that interval.

Could be either.

Ok, I understand now.

Right, and how do I adjust this in the code? Could you help me, please?

Thanks!

Updating your code is best done by yourself. Iâ€™m not familiar with your project code, and as a volunteer, Iâ€™m not inclined to do it for you.

Well, I thought this was a help forum, and I didnâ€™t ask you to update my code, just to help me. I didnâ€™t understand the sentence about the code update is better if it is done by myself. If I donâ€™t understand, I should seek help, right? How am I going to solve something If I donâ€™t know what to do?

Itâ€™s ok if youâ€™re not familiar with the code and that youâ€™re not willing to help, thanks!

This is a forum for questions. Iâ€™m not available to work on your code.

Perhaps someone else from the community will be able to help.

We donâ€™t know that code either: itâ€™s something new you just presented to us. And I guess what you are saying is that you donâ€™t understand that code and youâ€™d like us to do the work to understand it and then explain it to you or perhaps even debug it for you. You may get lucky and there is someone listening here who is sufficiently curious about the topic that they are willing to do that work for you. Please realize that the mentors are just volunteers here. We donâ€™t get paid to answer questions. So when you ask a question about something that is beyond the scope of any of the courses here, we have a choice of whether we want to spend our time in that way.

You gave a link to the paper on which this work is based. Have you read the paper?

Yes, I know itâ€™s for questions, and automatically, itâ€™s a help forum. And again I say that there is no problem with you not being available to help me, I just found it strange the way you talked about how updating the code is better if done by myself, especially because if I knew how to update the code I wouldnâ€™t be asking for help. Thanks.

I took a quick look at the paper. I didnâ€™t read every line, but my take is that what they are saying is not that there is an optimal learning rate, but that they got better training results in fewer iterations by varying the LR in a range in a triangular pattern. Meaning that they raise it and lower it alternately. So itâ€™s unclear to me what they mean by that code implementing something from the paper to find an optimal learning rate.

But this was with less than 10 minutes of effort, so Iâ€™m very likely missing the point â€¦

â€śto do the workâ€ť? Look, Iâ€™m trying to get help, Iâ€™m not saying you have to help me. So does this mean that in your opinion, a person who wants help wants someone to do the work for them? I know that no one is paid here, and no one is obligated to help, Iâ€™m not complaining that theyâ€™re not helping me, Iâ€™m just commenting on the way Iâ€™m being answered to.

Of course, at some point did I say that someone is obligated to help me?

1 Like

Yes, the problem is not understand the paper, but the code that is based on the paper.