Rauseo, Elisa; Omer, Muhammad; Amir-Khalili, Alborz; Sojoudi, Alireza; Le, Thu-Thao; Cook, Stuart Alexander; Hausenloy, Derek John; ... Petersen, Steffen Erhard; + view all Rauseo, Elisa; Omer, Muhammad; Amir-Khalili, Alborz; Sojoudi, Alireza; Le, Thu-Thao; Cook, Stuart Alexander; Hausenloy, Derek John; Ang, Briana; Toh, Desiree-Faye; Bryant, Jennifer; Chin, Calvin Woon Loong; Paiva, Jose Miguel; Fung, Kenneth; Cooper, Jackie; Khanji, Mohammed Yunus; Aung, Nay; Petersen, Steffen Erhard; - view fewer (2021) A Systematic Quality Scoring Analysis to Assess Automated Cardiovascular Magnetic Resonance Segmentation Algorithms. Frontiers in Cardiovascular Medicine , 8 , Article 816985. 10.3389/fcvm.2021.816985.

Preview

Text Hausenloy_A Systematic Quality Scoring Analysis to Assess Automated Cardiovascular Magnetic Resonance Segmentation Algorithms_VoR.pdf - Published Version Download (2MB) | Preview

Abstract

BACKGROUND: The quantitative measures used to assess the performance of automated methods often do not reflect the clinical acceptability of contouring. A quality-based assessment of automated cardiac magnetic resonance (CMR) segmentation more relevant to clinical practice is therefore needed. OBJECTIVE: We propose a new method for assessing the quality of machine learning (ML) outputs. We evaluate the clinical utility of the proposed method as it is employed to systematically analyse the quality of an automated contouring algorithm. METHODS: A dataset of short-axis (SAX) cine CMR images from a clinically heterogeneous population (n = 217) were manually contoured by a team of experienced investigators. On the same images we derived automated contours using a ML algorithm. A contour quality scoring application randomly presented manual and automated contours to four blinded clinicians, who were asked to assign a quality score from a predefined rubric. Firstly, we analyzed the distribution of quality scores between the two contouring methods across all clinicians. Secondly, we analyzed the interobserver reliability between the raters. Finally, we examined whether there was a variation in scores based on the type of contour, SAX slice level, and underlying disease. RESULTS: The overall distribution of scores between the two methods was significantly different, with automated contours scoring better than the manual (OR (95% CI) = 1.17 (1.07–1.28), p = 0.001; n = 9401). There was substantial scoring agreement between raters for each contouring method independently, albeit it was significantly better for automated segmentation (automated: AC2 = 0.940, 95% CI, 0.937–0.943 vs manual: AC2 = 0.934, 95% CI, 0.931–0.937; p = 0.006). Next, the analysis of quality scores based on different factors was performed. Our approach helped identify trends patterns of lower segmentation quality as observed for left ventricle epicardial and basal contours with both methods. Similarly, significant differences in quality between the two methods were also found in dilated cardiomyopathy and hypertension. CONCLUSIONS: Our results confirm the ability of our systematic scoring analysis to determine the clinical acceptability of automated contours. This approach focused on the contours' clinical utility could ultimately improve clinicians' confidence in artificial intelligence and its acceptability in the clinical workflow

Download activity - last month

Export as JSONCSVXML

Download activity - last 12 months

Export as XMLCSVJSON

Downloads by country - last 12 months

Export as JSONCSVXML

Archive Staff Only

View Item