The test power : the probability of detecting that difference between the original rate and the variant conversion rates.Minimum detectable effect : The desired relevant difference between the rates you would like to discover.It also means that you have significant result difference between the control and the variation with a 95% “confidence.” This threshold is, of course, an arbitrary one and one chooses it when making the design of an experiment. Using the statistical analysis of the results, you might reject or not reject the null hypothesis. There is a difference between the two conversion rates but you don’t have enough sample size (power) to detect it.The difference between the two conversion rates is too small to be relevant.There is no difference between the two conversion rates of the control and the variation (they are EXACTLY the same!).Not rejecting the null hypothesis means one of three things: Rejecting the null hypothesis means your data shows a statistically significant difference between the two conversion rates. The first case is very rare since the two conversion rates are usually different. The second case is ok since we are not interested in the difference which is less than the threshold we established for the experiment (like 0.01%). The worst case scenario is the third one. You are not able to detect a difference between the two conversion rates although it exists. Because of the data, you are completely unaware of it. To prevent this problem from happening, you need to calculate the sample size of your experiment before conducting it. It is important to remember that there is a difference between the population conversion rates and the sample size conversion observed rates r.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |