Christodoulou, Evangelia;
Reinke, Annika;
Houhou, Rola;
Kalinowski, Piotr;
Erkan, Selen;
Sudre, Carole H;
Burgos, Ninon;
... Maier-Hein, Lena; + view all
(2024)
Confidence Intervals Uncovered: Are We Ready for Real-World Medical Imaging AI?
In: Linguraru, MG and Dou, Q and Feragen, A and Giannarou, S and Glocker, B and Lekadir, K and Schnabel, JA, (eds.)
Medical Image Computing and Computer Assisted Intervention – MICCAI 2024.
(pp. pp. 124-132).
Springer: Cham, Switzerland.
Preview |
Text
2409.17763v2.pdf - Accepted Version Download (4MB) | Preview |
Abstract
Medical imaging is spearheading the AI transformation of healthcare. Performance reporting is key to determine which methods should be translated into clinical practice. Frequently, broad conclusions are simply derived from mean performance values. In this paper, we argue that this common practice is often a misleading simplification as it ignores performance variability. Our contribution is threefold. (1) Analyzing all MICCAI segmentation papers (n = 221) published in 2023, we first observe that more than 50% of papers do not assess performance variability at all. Moreover, only one (0.5%) paper reported confidence intervals (CIs) for model performance. (2) To address the reporting bottleneck, we show that the unreported standard deviation (SD) in segmentation papers can be approximated by a second-order polynomial function of the mean Dice similarity coefficient (DSC). Based on external validation data from 56 previous MICCAI challenges, we demonstrate that this approximation can accurately reconstruct the CI of a method using information provided in publications. (3) Finally, we reconstructed 95% CIs around the mean DSC of MICCAI 2023 segmentation papers. The median CI width was 0.03 which is three times larger than the median performance gap between the first and second ranked method. For more than 60% of papers, the mean performance of the second-ranked method was within the CI of the first-ranked method. We conclude that current publications typically do not provide sufficient evidence to support which models could potentially be translated into clinical practice.
| Type: | Proceedings paper |
|---|---|
| Title: | Confidence Intervals Uncovered: Are We Ready for Real-World Medical Imaging AI? |
| Event: | 27th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) |
| Location: | MOROCCO, Palmeraie Conf Ctr, Marrakesh |
| Dates: | 6 Oct 2024 - 10 Oct 2024 |
| ISBN-13: | 978-3-031-72116-8 |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1007/978-3-031-72117-5_12 |
| Publisher version: | https://doi.org/10.1007/978-3-031-72117-5_12 |
| Language: | English |
| Additional information: | This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions. |
| Keywords: | Clinical Translation, Computer Science, Computer Science, Artificial Intelligence, Computer Science, Theory & Methods, Confidence Intervals, Life Sciences & Biomedicine, Medical image segmentation, Radiology, Nuclear Medicine & Medical Imaging, Science & Technology, Technology, Variability reporting |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Population Health Sciences > Institute of Cardiovascular Science UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Population Health Sciences > Institute of Cardiovascular Science > Population Science and Experimental Medicine UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Population Health Sciences > Institute of Cardiovascular Science > Population Science and Experimental Medicine > MRC Unit for Lifelong Hlth and Ageing |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10205944 |
Archive Staff Only
 |
View Item |
