Methods: We obtained the signal-to-noise ratio distribution in phase III oncology using outcomes from 632 two-arm superiority trials enrolling 496,219 patients. With this distribution, we estimated successful replication probability as the probability that a replicate trial, having the same design, effect size, and standard error, would have a two-sided P ≤ 0.05 and the same effect directionality as the original trial. We also estimated the following: the probability that the estimated effect had the same direction as the true effect (i.e., correct sign probability); the probability that the 95 % CI covered the true effect (i.e., coverage probability), and the ratio of the observed estimated effect to the true effect (i.e., exaggeration factor).
Results: The median exaggeration factor across all trials was 1.09 (IQR, 0.80-1.61). When P ≤ 0.05 in the original trial, mean correct sign probabilities were ≥ 97 % and mean coverage probabilities were between 93 % and 96 %. However, effects at P of 0.05, 0.01, and 0.001 had mean replication probabilities of 43 % (95 % CI: 35-45 %), 60 % (95 % CI: 53-61 %), and 77 % (95 % CI: 71-79 %), respectively. For trials with an overall survival primary endpoint that led directly to regulatory approval, the median replication probability was 66 %. A user-friendly web interface is provided to facilitate estimation of replication probabilities of individual trials.
Conclusions: While the direction of observed effects is likely correct when P ≤ 0.05, treatment effects at P of 0.05 in phase III oncology trials are unlikely to be replicated successfully. By itself, statistical significance should not be equated with high replication probability.
Alexander D Sherry,1 Pavlos Msaouel,2 Avital M Miller,3 Timothy A Lin,3 Joseph Abi Jaoude,4 Ramez Kouzy,3 Adina H Passy,3 Tomer Meirson,5 Nikolaos Ignatiadis,6 Zachary R McCaw,7 Erik van Zwet,8 Ethan B Ludmir9
- Department of Radiation Oncology, Division of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA; Department of Radiation Oncology, Mayo Clinic, Rochester, MN, USA.
- Department of Genitourinary Medical Oncology, Division of Cancer Medicine, The University of Texas MD Anderson Cancer Center, Houston, TX, USA; Department of Translational Molecular Pathology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA.
- Department of Radiation Oncology, Division of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA.
- Department of Radiation Oncology, Stanford University, Stanford, CA, USA.
- Davidoff Cancer Center, Rabin Medical Center-Beilinson Hospital, Petach Tikva, Israel.
- Department of Statistics and Data Science Institute, University of Chicago, Chicago, IL, USA.
- Insitro, South San Francisco, CA, USA; Department of Biostatistics, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA.
- Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands.
- Department of Gastrointestinal Radiation Oncology, Division of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA; Department of Biostatistics, The University of Texas MD Anderson Cancer Center, Houston, TX, USA.
Go Beyond the Abstract and Read a Commentary by the Authors