Carmichael, Josie;
Costanza, Enrico;
Blandford, Ann;
Struyven, Robbert;
Keane, Pearse A;
Balaskas, Konstantinos;
(2024)
Diagnostic decisions of specialist optometrists exposed to ambiguous deep-learning outputs.
Scientific Reports
, 14
, Article 6775. 10.1038/s41598-024-55410-0.
Preview |
Text
s41598-024-55410-0.pdf - Published Version Download (2MB) | Preview |
Abstract
Artificial intelligence (AI) has great potential in ophthalmology. We investigated how ambiguous outputs from an AI diagnostic support system (AI-DSS) affected diagnostic responses from optometrists when assessing cases of suspected retinal disease. Thirty optometrists (15 more experienced, 15 less) assessed 30 clinical cases. For ten, participants saw an optical coherence tomography (OCT) scan, basic clinical information and retinal photography (‘no AI’). For another ten, they were also given AI-generated OCT-based probabilistic diagnoses (‘AI diagnosis’); and for ten, both AI-diagnosis and AI-generated OCT segmentations (‘AI diagnosis + segmentation’) were provided. Cases were matched across the three types of presentation and were selected to include 40% ambiguous and 20% incorrect AI outputs. Optometrist diagnostic agreement with the predefined reference standard was lowest for ‘AI diagnosis + segmentation’ (204/300, 68%) compared to ‘AI diagnosis’ (224/300, 75% p = 0.010), and ‘no Al’ (242/300, 81%, p = < 0.001). Agreement with AI diagnosis consistent with the reference standard decreased (174/210 vs 199/210, p = 0.003), but participants trusted the AI more (p = 0.029) with segmentations. Practitioner experience did not affect diagnostic responses (p = 0.24). More experienced participants were more confident (p = 0.012) and trusted the AI less (p = 0.038). Our findings also highlight issues around reference standard definition.
| Type: | Article |
|---|---|
| Title: | Diagnostic decisions of specialist optometrists exposed to ambiguous deep-learning outputs |
| Location: | England |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1038/s41598-024-55410-0 |
| Publisher version: | http://dx.doi.org/10.1038/s41598-024-55410-0 |
| Language: | English |
| Additional information: | This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. |
| Keywords: | Ophthalmology, OCT, Retinal disease, Artifcial intelligence, Human–computer interaction |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > Div of Psychology and Lang Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > Institute of Ophthalmology UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > Div of Psychology and Lang Sciences > UCL Interaction Centre UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Med Phys and Biomedical Eng |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10190316 |
Archive Staff Only
 |
View Item |
