Null hypothesis/ Randomisation/ P values/ Statistical significance/ Clinical significance.
This summary sheet steps through a fictional randomised control trial (RCT) to demystify key research concepts and explain why they are seen as important. This includes: a null hypothesis, randomisation, P values, statistical significance and clinical significance.
Hypothetical situation: Midwives from a training course in acupuncture report back to you that they find acupuncture “like magic” for women presenting with early onset mastitis. They are noticing women experience a decrease in their temperature and immediate symptom relief. You decide to study this for your Masters project. To begin, you set your research question and a null hypothesis. The idea of using a null hypothesis is that you start with the assumption that there will be no differences between your groups. Then if you do find a difference you can use statistics to determine if that difference is real, or if it’s just a random, chance event.
Your research questions is: What is the effect of acupuncture use in early onset mastitis? Your null hypothesis is that there will be no difference in oral temperatures between group A and group B ½ hour following acupuncture treatment in Group A. After discussion with a statistician you decide to run a randomised trial with 30 women in both groups: an acupuncture group and a usual care group.
Randomisation is important because it is seen as a way of eliminating potential bias (that is, that your results could be due to some other event). By randomising women to a treatment and control group you reduce your chance of having an unknown viable responsible for your results. If you planned your study where women could self-select to receive acupuncture, it could be that the type of person choosing acupuncture also has other factors in their life, for example they might eat more fruit and vegetables, and it is this extra nutrition, not your treatment giving you the positive results. Through randomising you ideally will have an equal mix of women with a healthier lifestyle in both groups. This is why when discussing a trial that has not been randomised it’s important to make it clear that although the findings are interesting, further randomised controlled trials are required to verify the findings.
Hypothetical situation: You obtain the necessary ethics permission and successfully run your trial. Your findings are that more women having the acupuncture have a reduced temperature than the control. This could be good news!
However, how do you know if your findings reflect something valuable or a random event? Statistics can be used to calculate a P value. This will tell you how likely it is that your results are due to chance. It’s not possible to be 100% sure, but by setting a P value of 0.05 you are saying that there is a 5% chance, or a 1 in 20 chance that your results are just a random event. In other words if your treatment made no difference (the null hypothesis was true) 20 identical studies would give you findings where 19 would be negative and one positive. Therefore positive findings that have a P value equal to or less than 0.05 are unlikely to be due to chance. Although it’s always good to keep in mind that statistics means never having to say you are certain! It is of course possible that your findings reflect that one in 20 random event. A P value of 0.05 is usually accepted as the level required for acupuncture trials, however, if you were conducting a trial requiring a higher level of evidence, for example a costly new chemotherapy medication with potential serious side effects, a higher p value, for example <0.001(or 1 in a 1000), may be set.
Hypothetical situation: Your results come back with a P value of 0.02. You have demonstrated statistical significance. This could be good news!
However are your findings clinically significant? It may be that the relief obtained is temporary - two hours after treatment the women’s symptoms return and they require antibiotics. Although it sounds obvious it’s always a good to check out that reported statically significant findings are clinically significant. There would be little point discussing any research paper with midwives or obstetricians unless the results were also clinically significant.
Hypothetical situation: You collected secondary outcomes that examined symptom relief at three days and one week following treatment as well as antibiotic use. These findings demonstrate treatment provided statistical significance for lasting symptom relief and reduced use of antibiotics. Your results are now statistical significant and clinically significant. This could be good news!
However, can you be sure that your results would apply to a larger population of women? This is where confidence intervals (CI) give you further information. They show you the range that the treatment effect is likely to lie. If the confidence interval is narrow you can be quite confident that the estimate of the true effect is quite precise. However if the range is wide you can assume that the study was probably quite small. This means the trial provides less trustworthy information even though your finding was statically significant. It may be helpful to consider it like this: if you ask 30 people which political party will win the next general election how confident can you be that your findings will be correct on Election Day? Obviously if you ask 1000 people you will be more confident. Unfortunately as most acupuncture trials have small numbers of participants they have wide confidence intervals. It is therefore important when discussing these trials to not over claim the results, adding that due to the wide confidence intervals further research is required.
Hypothetical situation: Your pilot trial is published and generates interest form a funding body. A larger RCT is carried out with 2000 women. This demonstrates statistical significance with narrow confidence intervals for symptom reduction and less antibiotic use in women receiving acupuncture. Great news!
Obviously while these latest hypothetical results are ideal and would present a convincing case for the use of acupuncture, they are difficult to achieve in the real world. In reality no trial is prefect and you will see the limitations of a study discussed in the paper. It is important to be aware of these limitations so that you can present the research in a balanced way. Sometimes you may even want to draw people’s attention to a trial where the outcome was not positive for acupuncture- as it may be more important to demonstrate that acupuncture is a safe treatment to be using. While research often brings up more questions than it answers, and does not always present us with the findings we expect, it can be a very useful tool to open up conversations with western medical practitioners. These are important conversations have the potential to raise the awareness of acupuncture and to create referral pathways.