Huebner, GM;
Nicolson, ML;
Fell, MJ;
Kennard, H;
Elam, S;
Hanmer, C;
Johnson, C;
Are we heading towards a replicability crisis in energy efficiency research? A toolkit for improving the quality, transparency and replicability of energy efficiency impact evaluations.
In:
Proceedings of the European Council for an Energy Efficient Economy ECEEE 2017 Summer Study on energy efficiency: consumption, efficiency and limits.
UKERC: London, UK.
Preview |
Text
Elam_8-299-17_Huebner_eceee_2017_final.pdf - Accepted Version Download (320kB) | Preview |
Abstract
Several high-profile replication failures have called into question the reproducibility of results in medicine, neuroscience, genetics, psychology and economics (Camerer et al. 2016). A paper published in Science found that just one third of psychology studies could be replicated when the study was run for a second time (OSC 2015). To our knowledge, there have been no attempted replications of energy efficiency studies; so can we be confident that the estimated energy savings from policy initiatives like the European roll out of smart meters will be realised? Or that electric vehicles will reduce carbon emissions by predicted levels? Or is energy heading towards its own reproducibility crisis? Researchers call for the increased use of randomised control trials (RCTs) to evaluate energy efficiency policy and the introduction of protocols or guidelines for conducting experiments (Vine et al. 2014; Frederiks et al. 2016). However, no guidelines for increasing reproducibility have been proposed. Moreover, RCTs are just one method for causal analysis and RCTs cannot answer all important causal questions. This paper will outline research methods for improved impact assessment of energy efficiency policy, including RCTs, but also quasi-experiments and systematic reviews that go beyond the conclusions of single experiments. It will then present tools for increasing replicability: pre-registration of trials; pre-analysis plans; reporting standards; synthesis tools and; publication of datasets with computer code in data repositories. Based on work by our research group at the UCL Energy Institute, we recognize that not all of these tools (mostly from medical trials) provide ‘off-the-shelf’ models for energy efficiency evaluations, and so consider adaptations for energy research. Our aim is to stimulate discussion and get feedback from the research community at ECEEE so the toolkit can be developed and potentially adopted more widely.
Type: | Proceedings paper |
---|---|
Title: | Are we heading towards a replicability crisis in energy efficiency research? A toolkit for improving the quality, transparency and replicability of energy efficiency impact evaluations |
Event: | European Council for an Energy Efficient Economy ECEEE 2017 Summer Study on energy efficiency: consumption, efficiency and limits |
Location: | Hyeres, France |
Dates: | 29 May 2017 - 03 June 2017 |
Open access status: | An open access version is available from UCL Discovery |
Publisher version: | http://www.ukerc.ac.uk/events/eceee-2017-summer-st... |
Language: | English |
Additional information: | This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions. |
UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of the Built Environment UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of the Built Environment > Bartlett School Env, Energy and Resources |
URI: | https://discovery.ucl.ac.uk/id/eprint/1561512 |
Archive Staff Only
View Item |