Use p-value and confidence intervals to analyze your data when using the A/B view.
The p-value represents the statistical confidence of the results. This tells you how likely the observed difference between the groups is due to chance. The initial hypothesis, or assumption, is that the two groups are not different. The lower the p-value, the more “real” or statistically significant the difference.
Interana allows you to set the statistical significance cutoff for your test. Use the following p-value settings to configure the data displays:
- 0.05: This is the standard measure of statistical significance, indicating that there is a 95% probability that the two groups are distinct in the selected measure. This is the default value.
- 0.01: This setting shows data with a higher statistical certainty, indicating a 99% probability that the two groups are distinct in the selected measure.
- 0.001: This setting shows data with an even greater level of statistical certainty.
This setting sets the significance cutoff for all applicable measures in the view.
Interana displays confidence intervals only when using the average measurement or certain custom ratio measurements. The confidence interval represents the range of values that our measure likely lies within, to the probability defined by our p-value (the p-val setting controls the width of the bands).
For example, with p-value setting of 0.05, we can be 95% confident that the actual difference between groups A and B for average itemInSession is 1.69 +/- 0.11, or between 1.58 and 1.80.
Use the confidence intervals to better understand when the difference between groups is statistically significant. The intervals show the distribution of values over all events.