Youyou, Wu;
Yang, Yang;
Uzzi, Brian;
(2023)
A discipline-wide investigation of the replicability of Psychology papers over the past two decades.
Proceedings of the National Academy of Sciences (PNAS)
, 120
(6)
, Article e2208863120. 10.1073/pnas.2208863120.
Preview |
Text
pnas.2208863120.pdf - Published Version Download (383kB) | Preview |
Abstract
Conjecture about the weak replicability in social sciences has made scholars eager to quantify the scale and scope of replication failure for a discipline. Yet small-scale manual replication methods alone are ill-suited to deal with this big data problem. Here, we conduct a discipline-wide replication census in science. Our sample (N = 14,126 papers) covers nearly all papers published in the six top-tier Psychology journals over the past 20 y. Using a validated machine learning model that estimates a paper's likelihood of replication, we found evidence that both supports and refutes speculations drawn from a relatively small sample of manual replications. First, we find that a single overall replication rate of Psychology poorly captures the varying degree of replicability among subfields. Second, we find that replication rates are strongly correlated with research methods in all subfields. Experiments replicate at a significantly lower rate than do non-experimental studies. Third, we find that authors' cumulative publication number and citation impact are positively related to the likelihood of replication, while other proxies of research quality and rigor, such as an author's university prestige and a paper's citations, are unrelated to replicability. Finally, contrary to the ideal that media attention should cover replicable research, we find that media attention is positively related to the likelihood of replication failure. Our assessments of the scale and scope of replicability are important next steps toward broadly resolving issues of replicability.
| Type: | Article |
|---|---|
| Title: | A discipline-wide investigation of the replicability of Psychology papers over the past two decades |
| Location: | United States |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1073/pnas.2208863120 |
| Publisher version: | https://doi.org/10.1073/pnas.2208863120 |
| Language: | English |
| Additional information: | Copyright © 2023 the Author(s). Published by PNAS. This open access article is distributed under Creative Commons Attribution-NonCommercial-NoDerivatives License 4.0 (CC BY-NC-ND). |
| Keywords: | Science of science, replication, machine learning, psychology |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > School of Education UCL > Provost and Vice Provost Offices > School of Education > UCL Institute of Education UCL > Provost and Vice Provost Offices > School of Education > UCL Institute of Education > IOE - Psychology and Human Development |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10164573 |
Archive Staff Only
 |
View Item |
