Key Takeaways
- Researchers urge a shift from viewing “non-significant” results as evidence of no effect.
- Equivalence testing can better capture nuanced scientific findings, avoiding oversimplified conclusions.
- A free online calculator is now available to facilitate equivalence testing for researchers.
Scientists from the Universities of Manchester, Oxford, and Arkansas are challenging the widespread notion that “non-significant” statistical results imply that no effect exists. In a recent paper published in the prestigious journal PNAS, the authors highlight a critical statistical error often made in research, which could lead to erroneous conclusions drawn from data.
The traditional p-value threshold of 0.05 is commonly interpreted as indicating that if a result is above this level, there is no effect. However, the researchers argue that a p-value greater than this simply indicates inadequate evidence to conclude that a difference exists. In about half of research papers and presentations, this misinterpretation poses a risk of neglecting important findings.
According to the authors, a non-significant result may arise from either an actual lack of effect or from underlying issues such as small sample sizes or high variability in the data. Thus, reducing complex scientific findings to a simplistic binary conclusion risks ignoring meaningful effects that could enhance understanding in various fields.
Many studies do not have sufficient sample sizes to detect true differences. Incorrectly reporting that no effect exists can prevent the identification of promising therapies or genuine risks, ultimately hindering follow-up research in those areas. To mitigate this issue, the authors advocate for a statistical approach known as equivalence testing, which focuses on identifying whether any existing difference is too trivial to be of scientific, clinical, or practical significance.
Equivalence testing allows researchers to discern between genuinely negligible effects and inconclusive results caused by insufficient evidence. One prominent method is the two one-sided tests procedure (TOST), gaining traction in psychology, medicine, and pharmaceutical regulation but still underutilized in other scientific disciplines.
Implementing equivalence testing more broadly could enhance scientific reporting quality and prevent misinterpretation of non-significant findings. David Eisner, a co-author and Professor of Cardiac Physiology at The University of Manchester, emphasized the importance of distinguishing between absence of effect and levels of significance when interpreting research findings.
Jakub Tomek from the University of Oxford noted that equivalence testing clarifies the data narrative, enabling better understanding and confidence in reported conclusions. The authors aim to promote careful interpretation of research findings, thereby influencing better scientific communication and application.
To make equivalence testing more accessible, the researchers developed a free online calculator that enables users to perform common tests without needing to write computer code. This tool supports various statistical comparisons and has been validated against established software.
Aaron Caldwell from the University of Arkansas for Medical Sciences stressed that equivalence testing compels researchers to consider a vital question: “How small is small enough to be uninteresting?” Making such a judgment calls for a scientific, rather than purely statistical, perspective and should occur prior to data collection. This approach allows for affirmative statements regarding absence of effects rather than merely non-detection.
For further reading, visit the source from the University of Manchester.
The content above is a summary. For more details, see the source article.