Kusa, Wojciech;
Lipani, Aldo;
Knoth, Petr;
Hanbury, Allan;
(2023)
An Analysis of Work Saved over Sampling in the Evaluation of Automated Citation Screening in Systematic Literature Reviews.
Intelligent Systems with Applications
, Article 200193. 10.1016/j.iswa.2023.200193.
(In press).
Preview |
PDF
1-s2.0-S2667305323000182-main.pdf - Accepted Version Download (999kB) | Preview |
Abstract
Citation screening is an essential and time-consuming step of the systematic literature review process in medicine. Multiple previous studies have proposed various automation techniques to assist manual annotators in this tedious task. The most widely used measure for the evaluation of automated citation screening techniques is Work Saved over Sampling (WSS). In this work, we analyse this measure and examine its drawbacks. We subsequently propose to normalise WSS which enables citation screening performance comparisons across different systematic reviews. We analytically show that normalised WSS is equivalent to the True Negative Rate (TNR). Finally, we provide benchmark scores for fifteen systematic review datasets with TNR@95% recall measure and compare the measure with Precision and AUC.
| Type: | Article |
|---|---|
| Title: | An Analysis of Work Saved over Sampling in the Evaluation of Automated Citation Screening in Systematic Literature Reviews |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1016/j.iswa.2023.200193 |
| Publisher version: | https://doi.org/10.1016/j.iswa.2023.200193 |
| Language: | English |
| Additional information: | © 2021 Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. |
| Keywords: | Citation Screening, Study Selection, Evaluation, Work Saved over Sampling (WSS), Systematic Literature Review (SLR) |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Civil, Environ and Geomatic Eng |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10164959 |
Archive Staff Only
 |
View Item |
