Week2-How to interpret these charts when traffic is shifted to one variant?

As only variantB is getting traffic (“DesiredWeight”: 100) and variantA does not get any traffic(“DesiredWeight”: 0), I had expected the charts to look different from what I see.
Please help me understand what do these charts tell?


1 Like

Hi Madhu!

Welcome to discourse! Let me try to explain:

You can see exactly that in all time series, a point in which B gets 100% (~12.58). Exception is model latency which does not apply as it stopped getting demands.

I hope that helps!


Thanks Raul! I got it. However, just
wondering if it is reasonable to see a lag of around 16minutes (start is 12:42) for our desired change (shift in traffic) to be effective?

@Madhu: from what I understand this period depends on the models you may choose to compare performance metrics and also reaching a sample enough to conclude which is the best one.

The lag is actually when Sagemaker starts to channel more traffic to the best model, around 12:54-55 timestamps in your charts. Before that point, it is the time dedicated to A/B testing models’ predictions (lines glued together).

I hope this explanation helps you understanding what is running under-the-hood. :wink: