Please correct content Error in A/B Testing video@7:41min

Course Q&A Math for Machine Learning Probability & Statistics for Machine Learning &...
1

Screen Shot 2023-06-29 at 11.36.50 PM
Screen Shot 2023-06-29 at 11.36.50 PM1744×976 93.7 KB

The p-value was found to be: p-value = 0.091

P-value = 0.091 > alpha = 0.05

The video is showing “Do NOT Reject H0”.

But that narration is saying: Reject H0 and accept H1 due to p-value = 0.019

Lecture transcription:

“What is the p-value for the sample? Well, it will be the area under this Gaussian PDF which has a value of 0.019. Since this is smaller than the significance level, you can reject the null hypothesis and accept that as true, that the conversion rate of Design B is bigger than the one for Design A.”

1 Like
2

I agree this slide is more than a confusing. First of all, z value is indeed -1.336.

001
0011148×590 280 KB

Now, if one calculates area under the Gaussian standardized curve from -Inf to -1.336, it is indeed 0.091. Around this time, instructor mentions that observed statistics is -2.07, and it is not clear how!
Very confusing indeed. It seems that p value is correctly calculated for the input data and it is 0.091, so there is not enough evidence to reject the null hypothesis H0.

3

Hey @Kevin_Shey and @Askaleto,
Thanks a lot for pointing out the discrepancies in the lecture video. To these, I would like to add a few more discrepancies that are present in the lecture video.

At 4:50, Luis states the following:

X/n_A has a mean P_A and a standard deviation, square root of P_A1-P_A/n_A. Y and B will have parameters Mu = P_B and Sigma equals square root of P_B1-P_B/n_B.

However, in the slide, as per the convention followed in the previous lecture videos, we have denoted the normal distributions with the help of the variance. It would be great, if the transcription along with the verbal description can be corrected, to avoid any possible source of confusion.

At 5:07, Luis states the following:

That’s because this is the standard deviation for the binomial distribution.

I believe it should be “variance” instead of “standard deviation”. Once again, please correct the audio + transcription here.

At 7:25, Luis states the following:

This means that the observed statistic is -2.07. What is the p-value for the sample? Well, it will be the area under this Gaussian PDF which has a value of 0.019.

As per the slide, observed statistic is -1.336. Please correct the audio + transcription here. Also, as Kevin and Asim have said, it should be 0.091 instead of 0.019. Additionally, the area marked in yellow is incorrect, and the label is also incorrect. Please correct the visualization here.

P.S. - Please unlist this thread, once the issues have been rectified.

Cheers,
Elemento

4

Glad I wasn’t the only one that got confused with the explanation…

Even in this situation, we say that Pa-Pb < 0. Rejecting H0 would thus imply H1, which in this case would mean that the conversion rate of Pa is lower than Pb (which I believe is what Luis wanted to say).

5

Thanks for noting this. I thought I was losing it. It would be good if they could just add a reading that had some of these discrepancies noted. Though it did give me a chance to use the discussion board.