UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

What is the suitability of clinical vignettes in benchmarking the performance of online symptom checkers? An audit study

El-Osta, Austen; Webber, Iman; Alaa, Aos; Bagkeris, Emmanouil; Mian, Saba; Sharabiani, Mansour; Majeed, Azeem; (2021) What is the suitability of clinical vignettes in benchmarking the performance of online symptom checkers? An audit study. MedRxiv: Cold Spring Harbor, NY, USA. Green open access

[thumbnail of Bagkeris_2021.07.29.21261320.full.pdf]
Preview
Text
Bagkeris_2021.07.29.21261320.full.pdf

Download (401kB) | Preview

Abstract

Objective: To assess the suitability of primary care vignettes in benchmarking the performance of online symptom checkers / Design: Observational study using publicly available, free online symptom checkers / Participants: Three symptom checkers (Healthily, Ada and Babylon) that provided consultations in English. 139 standardized patient vignettes were compiled by RCGP. Three independent GPs interpreted the vignettes to arrive at a “Gold Standard” consisting of 3 dispositions and divided into one of three categories of triage urgency: (1) emergency care required, (2) primary care required and (3) self-care. / Main outcome measures: Six professional non-medical and lay inputters simulated 2774 standardized patient evaluations using 3 online symptom checkers (OSC). We recorded when OSC provided a triage recommendation and whether it correctly recommended the appropriate triage recommendation across three categories of triage urgency (emergency care, primary care or self-care). We collected data on whether the solution appeared within the first 3 dispositions in each of the standards across 2774 standardized patient evaluations. / Results: When benchmarked against the Gold Standard, Healthily provided an appropriate triage recommendation 61.9% of the time compared to 45.3% and 42.4% of the time for Babylon and Ada respectively. There was poor agreement between OSC consultation outcome and Gold Standard dispositions. When compared to the Gold Standard, Healthily gave an unsafe “under-triage” recommendation 28.6% of the time overall across the three categories compared to 43.3% for Ada and 47.5% for Babylon (P<0.001). / Conclusions: OSCs recommended ‘very unsafe’ triages only <4% of the time suggesting that the online consultation tools are generally working at a safe level of risk. Primary care vignettes are a helpful tool to support development of OSC, but not ideally suited to benchmark the performance of different OSC. Real-world evidence studies involving general practice are recommended to benchmark the performance of OSC in the community setting. /

Type: Working / discussion paper
Title: What is the suitability of clinical vignettes in benchmarking the performance of online symptom checkers? An audit study
Open access status: An open access version is available from UCL Discovery
DOI: 10.1101/2021.07.29.21261320
Publisher version: https://doi.org/10.1101/2021.07.29.21261320
Language: English
Additional information: The copyright holder for this preprint is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY-NC-ND 4.0 International license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
UCL classification: UCL
UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Population Health Sciences > UCL GOS Institute of Child Health > Population, Policy and Practice Dept
UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Population Health Sciences > UCL GOS Institute of Child Health
UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences
URI: https://discovery.ucl.ac.uk/id/eprint/10147713
Downloads since deposit
14Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item