SIGNIFICANCE SYNDROME: A MIXED METHODS EXPLORATION OF P VALUE OVERRELIANCE IN MEDICAL SCIENCE

Abstract

Introduction: Over the past three decades, the p‑value has become the dominant metric for judging statistical significance in biomedical research, yet its binary interpretation (significant vs. non‑significant) has contributed to publication bias, irreproducible findings, and an undue focus on arbitrary thresholds.

Aim and Objectives: This mixed‑methods study was designed to examine longitudinal trends in p‑value reporting and to explore researchers’ attitudes toward p‑value usage and alternatives. Specifically, we aimed to (1) quantify the proportion of articles reporting the p‑value alone versus those also including effect sizes, confidence intervals, or alternative inferential metrics at four timepoints (1990, 2000, 2010, and 2020–2024); (2) identify study and journal characteristics associated with richer statistical reporting; and (3) assess researchers’ perceived pressure to achieve p<0.05 and their familiarity with non‑p‑value approaches.

Methods: We conducted a bibliometric analysis of 1,042 original research articles sampled from five high‑impact journals (JAMA, The Lancet, NEJM, BMJ, and PLOS Medicine) at the four specified timepoints. Each article was classified by reporting style—p‑value only; p‑value + effect size; p‑value + confidence interval; or alternative metrics—and data were extracted on study design, journal impact factor, funding source, author count, and geographic region. Concurrently, we administered a cross‑sectional online survey to 217 active biomedical researchers, capturing perceived publication pressure, statistical training background, and familiarity with effect sizes, confidence intervals, and Bayesian methods. Trends were analyzed using Cochran–Armitage tests, and multivariable logistic regression identified independent predictors of enriched reporting.

Results: Exclusive p‑value reporting declined from 76.3% in 1990 to 37.9% in 2020–2024. Articles in higher‑impact journals and randomized controlled trials were significantly more likely to include effect sizes and confidence intervals. Survey respondents reported high familiarity with effect sizes (92%) and confidence intervals (95%) but lower familiarity with Bayesian methods (42%), and 78% felt pressured to meet p<0.05 thresholds, particularly early‑career investigators without advanced statistical training.

Conclusion: Although editorial guidelines have encouraged more informative reporting, entrenched incentives and gaps in statistical literacy continue to drive “significance syndrome.” We recommend mandating effect‑size and interval reporting, integrating advanced inferential training into research curricula, and promoting preregistration and open data sharing to enhance transparency and reproducibility.