The same approach used to calculate a confidence interval for the effect size or statistical power of a t test (see Confidence Intervals for Power and Effect Size of t Tests) can be employed to create a confidence interval for the noncentrality parameter, and in turn the RMSSE effect size and statistical power, of one-factor ANOVA.
Example 1: Find the 95% confidence interval for RMSSE and power of the one-way ANOVA of Example 1 of Power of One-way ANOVA.
The analysis is shown in Figure 1 (with references to cells in Figure 1 of Power of One-way ANOVA).
Figure 1 – Confidence intervals for RMSSE and statistical power
We see from Figure 1 that the 95% confidence interval for the noncentrality parameter is (0, 23.68). The corresponding confidence interval for the RMSSE of .62 is (0, .99) and the confidence interval for power of 65.3% is (5%, 97.8%).
Even though this test results in a relatively high effect size (.62) and power of 65.3%, because the sample is very small, the confidence intervals are quite wide, and so if the test is repeated with similar sized data it is not so unlikely that the resulting effect size and power can be near zero. This reinforces what we should already know, namely that caution needs to be applied when running statistical tests, especially with small sample sizes.