UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

Are peer-reviews of grant proposals reliable? An analysis of Economic and Social Research Council (ESRC) funding applications

Jerrim, J; Vries, RD; (2020) Are peer-reviews of grant proposals reliable? An analysis of Economic and Social Research Council (ESRC) funding applications. Social Science Journal 10.1080/03623319.2020.1728506. (In press). Green open access

[thumbnail of Jerrim_Main_Body_November_2019.pdf]
Preview
Text
Jerrim_Main_Body_November_2019.pdf - Accepted Version

Download (697kB) | Preview

Abstract

Peer-review is widely used throughout academia, most notably in the publication of journal articles and the allocation of research grants. Yet peer-review has been subject to much criticism, including being slow, unreliable, subjective and potentially prone to bias. This paper contributes to this literature by investigating the consistency of peer-reviews and the impact they have upon a high-stakes outcome (whether a research grant is funded). Analysing data from 4,000 social science grant proposals and 15,000 reviews, this paper illustrates how the peer-review scores assigned by different reviewers have only low levels of consistency (a correlation between reviewer scores of only 0.2). Reviews provided by ‘nominated reviewers’ (i.e. reviewers selected by the grant applicant) appear to be overly generous and do not correlate with the evaluations provided by independent reviewers. Yet a positive review from a nominated reviewer is strongly linked to whether a grant is awarded. Finally, a single negative peer-review is shown to reduce the chances of a proposal being funding from around 55% to around 25% (even when it has otherwise been rated highly).

Type: Article
Title: Are peer-reviews of grant proposals reliable? An analysis of Economic and Social Research Council (ESRC) funding applications
Open access status: An open access version is available from UCL Discovery
DOI: 10.1080/03623319.2020.1728506
Publisher version: https://doi.org/10.1080/03623319.2020.1728506
Language: English
Additional information: This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions.
Keywords: Peer-review, consistency, grant funding
UCL classification: UCL
UCL > Provost and Vice Provost Offices
UCL > Provost and Vice Provost Offices > School of Education
UCL > Provost and Vice Provost Offices > School of Education > UCL Institute of Education
UCL > Provost and Vice Provost Offices > School of Education > UCL Institute of Education > IOE - Social Research Institute
URI: https://discovery.ucl.ac.uk/id/eprint/10095820
Downloads since deposit
156Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item