Understanding basic statistics is one of the most important skills to develop as you begin to optimize. Statistics provide insight into your results and give you confidence in your winning variations. Using statistical values in your decision-making leads to stable, replicable results you can bet your business on. On the other hand, a lack of understanding can lead to errors and unreliable outcomes from your experiments.
Let’s start with an example. Farmer Fred wants to compare the effect of two fertilizers on crop yield. He decides how many plots he’ll use for his experiment - this is his sample size. He then has to wait for a crop cycle and collects data at the end of that cycle. He asks, “what are the chances I would have gotten these same results if there was no difference between the fertilizers?” This is the P value. If the P-value is less than 5%, his results are significant. In this case, it means the results Farmer Fred observed are, in fact, probably due to a difference in the fertilizers he used. Now he can go on to test other fertilization methods.
But classical statistics were designed for an offline world! In the old analog days, data was expensive, slow, and practitioners had to be trained to use it. Now, data is cheap, gathered in real time, and everyone can learn to understand it.
Modern A/B testing is different from classical statistics. Ask yourself how your approach differs from Farmer Fred’s.
For one thing, you have to check your results early and often to estimate ROI as quickly as possible. You don’t have time to wait until the crop season is over. In the digital world, you may also have many goals and multiple variations at once. Farmer Fred’s single-goal experiment program just isn’t realistic anymore.
But patience is still important. Often, when you run an experiment, it’s tempting to check results repeatedly in search of a discovery. It’s also very tempting to stop an experiment when it reaches statistical significance the first time, even if that happens before reaching the required sample size. Be sure to allow enough time and get enough eyes on your experiment before you stop an experiment. Ending your experiment too soon could give you bad data and undo all your hard work up to this point.