UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

Quantifying “Promising Trials Bias” in Randomized Controlled Trials in Education

Sims, Sam; Anders, Jake; Inglis, Matthew; Lortie-Forgues, Hugues; (2022) Quantifying “Promising Trials Bias” in Randomized Controlled Trials in Education. Journal of Research on Educational Effectiveness pp. 1-18. 10.1080/19345747.2022.2090470. (In press). Green open access

[thumbnail of Anders_19345747.2022.2090470.pdf]
Preview
Text
Anders_19345747.2022.2090470.pdf

Download (2MB) | Preview

Abstract

Randomized controlled trials have proliferated in education, in part because they provide an unbiased estimator for the causal impact of interventions. It is increasingly recognized that many such trials in education have low power to detect an effect if indeed there is one. However, it is less well known that low powered trials tend to systematically exaggerate effect sizes among the subset of interventions that show promising results (p<α). We conduct a retrospective design analysis to quantify this bias across 22 such promising trials, finding that the estimated effect sizes are exaggerated by an average of 52% or more. Promising trial bias can be reduced ex-ante by increasing the power of the trials that are commissioned and guarded against ex-post by including estimates of the exaggeration ratio when reporting trial findings. Our results also suggest that challenges around implementation fidelity are not the only reason that apparently successful interventions often fail to subsequently scale up. Instead, the effect from the initial promising trial may simply be exaggerated.

Type: Article
Title: Quantifying “Promising Trials Bias” in Randomized Controlled Trials in Education
Open access status: An open access version is available from UCL Discovery
DOI: 10.1080/19345747.2022.2090470
Publisher version: https://doi.org/10.1080/19345747.2022.2090470
Language: English
Additional information: © 2022 The Author(s). Published with license by Taylor & Francis Group, LLC. This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives License (http://creativecommons.org/licenses/by-nc-nd/4.0/).
Keywords: Randomized controlled trials; Type M error; Type S error
UCL classification: UCL > Provost and Vice Provost Offices > School of Education > UCL Institute of Education
UCL > Provost and Vice Provost Offices > School of Education > UCL Institute of Education > IOE - Learning and Leadership > Centre for Education Policy and Equalising Opportunities
UCL > Provost and Vice Provost Offices > School of Education
UCL
UCL > Provost and Vice Provost Offices > School of Education > UCL Institute of Education > IOE - Learning and Leadership
URI: https://discovery.ucl.ac.uk/id/eprint/10152380
Downloads since deposit
0Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item