UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

Evidence of unexplained discrepancies between planned and conducted statistical analyses: a review of randomised trials

Cro, S; Forbes, G; Johnson, NA; Kahan, BC; (2020) Evidence of unexplained discrepancies between planned and conducted statistical analyses: a review of randomised trials. BMC Medicine , 18 , Article 137. 10.1186/s12916-020-01590-1. Green open access

[thumbnail of s12916-020-01590-1.pdf]
Preview
Text
s12916-020-01590-1.pdf - Published Version

Download (687kB) | Preview

Abstract

BACKGROUND: Choosing or altering the planned statistical analysis approach after examination of trial data (often referred to as 'p-hacking') can bias the results of randomised trials. However, the extent of this issue in practice is currently unclear. We conducted a review of published randomised trials to evaluate how often a pre-specified analysis approach is publicly available, and how often the planned analysis is changed. METHODS: A review of randomised trials published between January and April 2018 in six leading general medical journals. For each trial, we established whether a pre-specified analysis approach was publicly available in a protocol or statistical analysis plan and compared this to the trial publication. RESULTS: Overall, 89 of 101 eligible trials (88%) had a publicly available pre-specified analysis approach. Only 22/89 trials (25%) had no unexplained discrepancies between the pre-specified and conducted analysis. Fifty-four trials (61%) had one or more unexplained discrepancies, and in 13 trials (15%), it was impossible to ascertain whether any unexplained discrepancies occurred due to incomplete reporting of the statistical methods. Unexplained discrepancies were most common for the analysis model (n = 31, 35%) and analysis population (n = 28, 31%), followed by the use of covariates (n = 23, 26%) and the approach for handling missing data (n = 16, 18%). Many protocols or statistical analysis plans were dated after the trial had begun, so earlier discrepancies may have been missed. CONCLUSIONS: Unexplained discrepancies in the statistical methods of randomised trials are common. Increased transparency is required for proper evaluation of results.

Type: Article
Title: Evidence of unexplained discrepancies between planned and conducted statistical analyses: a review of randomised trials
Location: England
Open access status: An open access version is available from UCL Discovery
DOI: 10.1186/s12916-020-01590-1
Publisher version: https://doi.org/10.1186/s12916-020-01590-1
Language: English
Additional information: This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
Keywords: P-hacking, Randomised controlled trials, Statistical analysis, Statistical analysis plan, Transparency
UCL classification: UCL
UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences
UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Population Health Sciences > Inst of Clinical Trials and Methodology
UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Population Health Sciences > Inst of Clinical Trials and Methodology > MRC Clinical Trials Unit at UCL
URI: https://discovery.ucl.ac.uk/id/eprint/10101060
Downloads since deposit
0Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item