Why use an alpha for A/B testing in C3_W4_Assignment

I understand that an alpha of 0.05 makes sense to answer scientific questions. But here the variation is that the users will see the website with new background colors. So if the average session duration is higher but there is a 10% chance that the improvement is due to chance rather than that the new color makes a real difference. But why wouldn’t someone go with the change anyway since there is a 90% chance that it improves session durations. This reasonning seems to apply to any alpha less than 0.5

HI @toontalk, thanks for your post.

@a-zarta, can you give a hand here?


Hi @toontalk, from my experience, companies usually go with these very low significance levels because they want to be absolutely sure that the change is gonna represent an increase in whatever metric they are interested in. Usually increasing a conversion rate (or some other metric) even by a few units can represented a huge increase in revenue. In the assignment this situation is meant to be more of a pedagogical situation in which someone interested in AB testing wants to perform the process in a rigorous way but in reality if you where doing this for a small project/website you might even not do the hypothesis testing altogether because you can afford to be wrong and harm the metrics you are interested in.