UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

Are we heading towards a replicability crisis in energy efficiency research? A toolkit for improving the quality, transparency and replicability of energy efficiency impact evaluations

Huebner, GM; Nicolson, ML; Fell, MJ; Kennard, H; Elam, S; Hanmer, C; Johnson, C; Are we heading towards a replicability crisis in energy efficiency research? A toolkit for improving the quality, transparency and replicability of energy efficiency impact evaluations. In: Proceedings of the European Council for an Energy Efficient Economy ECEEE 2017 Summer Study on energy efficiency: consumption, efficiency and limits. UKERC: London, UK. Green open access

[thumbnail of Elam_8-299-17_Huebner_eceee_2017_final.pdf]
Preview
Text
Elam_8-299-17_Huebner_eceee_2017_final.pdf - Accepted Version

Download (320kB) | Preview

Abstract

Several high-profile replication failures have called into question the reproducibility of results in medicine, neuroscience, genetics, psychology and economics (Camerer et al. 2016). A paper published in Science found that just one third of psychology studies could be replicated when the study was run for a second time (OSC 2015). To our knowledge, there have been no attempted replications of energy efficiency studies; so can we be confident that the estimated energy savings from policy initiatives like the European roll out of smart meters will be realised? Or that electric vehicles will reduce carbon emissions by predicted levels? Or is energy heading towards its own reproducibility crisis? Researchers call for the increased use of randomised control trials (RCTs) to evaluate energy efficiency policy and the introduction of protocols or guidelines for conducting experiments (Vine et al. 2014; Frederiks et al. 2016). However, no guidelines for increasing reproducibility have been proposed. Moreover, RCTs are just one method for causal analysis and RCTs cannot answer all important causal questions. This paper will outline research methods for improved impact assessment of energy efficiency policy, including RCTs, but also quasi-experiments and systematic reviews that go beyond the conclusions of single experiments. It will then present tools for increasing replicability: pre-registration of trials; pre-analysis plans; reporting standards; synthesis tools and; publication of datasets with computer code in data repositories. Based on work by our research group at the UCL Energy Institute, we recognize that not all of these tools (mostly from medical trials) provide ‘off-the-shelf’ models for energy efficiency evaluations, and so consider adaptations for energy research. Our aim is to stimulate discussion and get feedback from the research community at ECEEE so the toolkit can be developed and potentially adopted more widely.

Type: Proceedings paper
Title: Are we heading towards a replicability crisis in energy efficiency research? A toolkit for improving the quality, transparency and replicability of energy efficiency impact evaluations
Event: European Council for an Energy Efficient Economy ECEEE 2017 Summer Study on energy efficiency: consumption, efficiency and limits
Location: Hyeres, France
Dates: 29 May 2017 - 03 June 2017
Open access status: An open access version is available from UCL Discovery
Publisher version: http://www.ukerc.ac.uk/events/eceee-2017-summer-st...
Language: English
Additional information: This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions.
UCL classification: UCL
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of the Built Environment
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of the Built Environment > Bartlett School Env, Energy and Resources
URI: https://discovery.ucl.ac.uk/id/eprint/1561512
Downloads since deposit
134Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item