Christodoulou, Evangelia; Reinke, Annika; Houhou, Rola; Kalinowski, Piotr; Erkan, Selen; Sudre, Carole H; Burgos, Ninon; ... Maier-Hein, Lena; + view all Christodoulou, Evangelia; Reinke, Annika; Houhou, Rola; Kalinowski, Piotr; Erkan, Selen; Sudre, Carole H; Burgos, Ninon; Boutaj, Sofiene; Loizillon, Sophie; Solal, Maelys; Rieke, Nicola; Cheplygina, Veronika; Antonelli, Michela; Mayer, Leon D; Tizabi, Minu D; Cardoso, M Jorge; Simpson, Amber; Jaeger, Paul F; Kopp-Schneider, Annette; Varoquaux, Gael; Colliot, Olivier; Maier-Hein, Lena; - view fewer (2024) Confidence Intervals Uncovered: Are We Ready for Real-World Medical Imaging AI? In: Linguraru, MG and Dou, Q and Feragen, A and Giannarou, S and Glocker, B and Lekadir, K and Schnabel, JA, (eds.) Medical Image Computing and Computer Assisted Intervention – MICCAI 2024. (pp. pp. 124-132). Springer: Cham, Switzerland.

Text 2409.17763v2.pdf - Accepted Version Access restricted to UCL open access staff until 4 October 2025. Download (4MB)

Abstract

Medical imaging is spearheading the AI transformation of healthcare. Performance reporting is key to determine which methods should be translated into clinical practice. Frequently, broad conclusions are simply derived from mean performance values. In this paper, we argue that this common practice is often a misleading simplification as it ignores performance variability. Our contribution is threefold. (1) Analyzing all MICCAI segmentation papers (n = 221) published in 2023, we first observe that more than 50% of papers do not assess performance variability at all. Moreover, only one (0.5%) paper reported confidence intervals (CIs) for model performance. (2) To address the reporting bottleneck, we show that the unreported standard deviation (SD) in segmentation papers can be approximated by a second-order polynomial function of the mean Dice similarity coefficient (DSC). Based on external validation data from 56 previous MICCAI challenges, we demonstrate that this approximation can accurately reconstruct the CI of a method using information provided in publications. (3) Finally, we reconstructed 95% CIs around the mean DSC of MICCAI 2023 segmentation papers. The median CI width was 0.03 which is three times larger than the median performance gap between the first and second ranked method. For more than 60% of papers, the mean performance of the second-ranked method was within the CI of the first-ranked method. We conclude that current publications typically do not provide sufficient evidence to support which models could potentially be translated into clinical practice.

Download activity - last month

Export as XMLCSVJSON

Download activity - last 12 months

Export as CSVXMLJSON

Downloads by country - last 12 months

Export as JSONCSVXML

Archive Staff Only

View Item