Stansfield, C;
Stokes, G;
Thomas, J;
(2022)
Applying machine classifiers to update searches: analysis from two case studies.
Research Synthesis Methods
, 13
(1)
pp. 121-133.
10.1002/jrsm.1537.
(In press).
Preview |
Text
jrsm.1537.pdf - Accepted Version Download (582kB) | Preview |
Abstract
Manual screening of citation records could be reduced by using machine classifiers to remove records of very low relevance. This seems particularly feasible for update searches, where a machine classifier can be trained from past screening decisions. However, feasibility is unclear for broad topics. To evaluate the performance and implementation of machine classifiers for update searches of public health research. Two case studies. The first study evaluates the impact of using different sets of training data on classifier performance, comparing recall and screening reduction with a manual screening ‘gold standard’. The second study uses screening decisions from a review to train a classifier that is applied to rank the update search results. A stopping threshold was applied in the absence of a gold standard. Time spent screening titles and abstracts of different relevancy-ranked records was measured. Results: Study one: Classifier performance varies according to the training data used; all custom-built classifiers had a recall above 93% at the same threshold, achieving screening reductions between 41% and 74%. Study two: applying a classifier provided a solution for tackling a large volume of search results from the update search, and screening volume was reduced by 61%. A tentative estimate indicates over 25 hours screening time was saved. Custom-built machine classifiers are feasible for reducing screening workload from update searches across a range of public health interventions, with some limitation on recall. Key considerations include selecting a training dataset, agreeing stopping thresholds and processes to ensure smooth workflows.
| Type: | Article |
|---|---|
| Title: | Applying machine classifiers to update searches: analysis from two case studies |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1002/jrsm.1537 |
| Publisher version: | https://doi.org/10.1002/jrsm.1537 |
| Language: | English |
| Additional information: | © 2021 The Authors. Research Synthesis Methods published by John Wiley & Sons Ltd. This is an open access article under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/). |
| Keywords: | Information retrieval; Supervised machine learning; Systematic Reviews as Topic; Update search |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > School of Education UCL > Provost and Vice Provost Offices > School of Education > UCL Institute of Education UCL > Provost and Vice Provost Offices > School of Education > UCL Institute of Education > IOE - Social Research Institute |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10138111 |
Loading...
Loading...Loading...Loading...Archive Staff Only
 |
View Item |
