Colquhoun, D;
(2017)
The reproducibility of research and the misinterpretation of pvalues.
Royal Society Open Science
, 4
(12)
10.1098/rsos.171085.

Text
171085.full.pdf  ["content_typename_Published version" not defined] Download (1MB)  Preview 
Abstract
We wish to answer this question: If you observe a ‘significant’ pvalue after doing a single unbiased experiment, what is the probability that your result is a false positive? The weak evidence provided by pvalues between 0.01 and 0.05 is explored by exact calculations of false positive risks. When you observe p = 0.05, the odds in favour of there being a real effect (given by the likelihood ratio) are about 3 : 1. This is far weaker evidence than the odds of 19 to 1 that might, wrongly, be inferred from the pvalue. And if you want to limit the false positive risk to 5%, you would have to assume that you were 87% sure that there was a real effect before the experiment was done. If you observe p = 0.001 in a wellpowered experiment, it gives a likelihood ratio of almost 100 : 1 odds on there being a real effect. That would usually be regarded as conclusive. But the false positive risk would still be 8% if the prior probability of a real effect were only 0.1. And, in this case, if you wanted to achieve a false positive risk of 5% you would need to observe p = 0.00045. It is recommended that the terms ‘significant’ and ‘nonsignificant’ should never be used. Rather, pvalues should be supplemented by specifying the prior probability that would be needed to produce a specified (e.g. 5%) false positive risk. It may also be helpful to specify the minimum false positive risk associated with the observed pvalue. Despite decades of warnings, many areas of science still insist on labelling a result of p < 0.05 as ‘statistically significant’. This practice must contribute to the lack of reproducibility in some areas of science. This is before you get to the many other wellknown problems, like multiple comparisons, lack of randomization and phacking. Precise inductive inference is impossible and replication is the only way to be sure. Science is endangered by statistical misunderstanding, and by senior people who impose perverse incentives on scientists.
Type:  Article 

Title:  The reproducibility of research and the misinterpretation of pvalues 
Open access status:  An open access version is available from UCL Discovery 
DOI:  10.1098/rsos.171085 
Publisher version:  http://doi.org/10.1098/rsos.171085 
Language:  English 
Additional information:  Copyright © 2017 The Authors. Published by the Royal Society under the terms of the Creative Commons Attribution License http://creativecommons.org/licenses/by/4.0/, which permits unrestricted use, provided the original author and source are credited. 
Keywords:  statistics, significance tests, null hypothesis tests, reproducibility, false positive risk 
UCL classification:  UCL > Provost and Vice Provost Offices UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Life Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Life Sciences > Div of Biosciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Life Sciences > Div of Biosciences > Neuro, Physiology and Pharmacology 
URI:  http://discovery.ucl.ac.uk/id/eprint/10045861 
Archive Staff Only
View Item 