Shemilt, I;
Arno, A;
Thomas, J;
Lorenc, T;
Khouja, C;
Raine, G;
Sutcliffe, K;
... Sowden, A; + view all
(2021)
Cost-effectiveness of Microsoft Academic Graph with machine learning for automated study identification in a living map of coronavirus disease 2019 (COVID-19) research.
Wellcome Open Research
, 6
, Article 210. 10.12688/wellcomeopenres.17141.1.
Preview |
Text
317e2d7c-44c1-43f8-ab09-b86adf98b057_17141_-_james_thomas.pdf - Published Version Download (989kB) | Preview |
Abstract
BACKGROUND: Conventionally, searching for eligible articles to include in systematic reviews and maps of research has relied primarily on information specialists conducting Boolean searches of multiple databases and manually processing the results, including deduplication between these multiple sources. Searching one, comprehensive source, rather than multiple databases, could save time and resources. Microsoft Academic Graph (MAG) is potentially such a source, containing a network graph structure which provides metadata that can be exploited in machine learning processes. Research is needed to establish the relative advantage of using MAG as a single source, compared with conventional searches of multiple databases. This study sought to establish whether: (a) MAG is sufficiently comprehensive to maintain our living map of coronavirus disease 2019 (COVID-19) research; and (b) eligible records can be identified with an acceptably high level of specificity. METHODS: We conducted a pragmatic, eight-arm cost-effectiveness analysis (simulation study) to assess the costs, recall and precision of our semi-automated MAG-enabled workflow versus conventional searches of MEDLINE and Embase (with and without machine learning classifiers, active learning and/or fixed screening targets) for maintaining a living map of COVID-19 research. Resource use data (time use) were collected from information specialists and other researchers involved in map production. RESULTS: MAG-enabled workflows dominated MEDLINE-Embase workflows in both the base case and sensitivity analyses. At one month (base case analysis) our MAG-enabled workflow with machine learning, active learning and fixed screening targets identified n=469 more new, eligible articles for inclusion in our living map – and cost £3,179 GBP ($5,691 AUD) less – than conventional MEDLINE-Embase searches without any automation or fixed screening targets. CONCLUSIONS: MAG-enabled continuous surveillance workflows have potential to revolutionise study identification methods for living maps, specialised registers, databases of research studies and/or collections of systematic reviews, by increasing their recall and coverage, whilst reducing production costs.
| Type: | Article |
|---|---|
| Title: | Cost-effectiveness of Microsoft Academic Graph with machine learning for automated study identification in a living map of coronavirus disease 2019 (COVID-19) research |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.12688/wellcomeopenres.17141.1 |
| Publisher version: | https://doi.org/10.12688/wellcomeopenres.17141.1 |
| Language: | English |
| Additional information: | © 2021 Shemilt I et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. |
| Keywords: | Automation, systematic review, machine learning, evidence synthesis, systematic map |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > School of Education UCL > Provost and Vice Provost Offices > School of Education > UCL Institute of Education UCL > Provost and Vice Provost Offices > School of Education > UCL Institute of Education > IOE - Social Research Institute |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10133853 |
Archive Staff Only
 |
View Item |
