Sirajudeen, N;
Boal, M;
Anastasiou, D;
Xu, J;
Stoyanov, D;
Kelly, J;
Collins, JW;
... Francis, NK; + view all
(2024)
Deep learning prediction of error and skill in robotic prostatectomy suturing.
Surgical Endoscopy and Other Interventional Techniques
, 38
(12)
pp. 7663-7671.
10.1007/s00464-024-11341-5.
Preview |
Text
Deep learning prediction of error and skill in robotic prostatectomy suturing.pdf - Published Version Download (818kB) | Preview |
Abstract
Background: Manual objective assessment of skill and errors in minimally invasive surgery have been validated with correlation to surgical expertise and patient outcomes. However, assessment and error annotation can be subjective and are time-consuming processes, often precluding their use. Recent years have seen the development of artificial intelligence models to work towards automating the process to allow reduction of errors and truly objective assessment. This study aimed to validate surgical skill rating and error annotations in suturing gestures to inform the development and evaluation of AI models. // Methods: SAR-RARP50 open data set was blindly, independently annotated at the gesture level in Robotic-Assisted Radical Prostatectomy (RARP) suturing. Manual objective assessment tools and error annotation methodology, Objective Clinical Human Reliability Analysis (OCHRA), were used as ground truth to train and test vision-based deep learning methods to estimate skill and errors. Analysis included descriptive statistics plus tool validity and reliability. // Results: Fifty-four RARP videos (266 min) were analysed. Strong/excellent inter-rater reliability (range r = 0.70–0.89, p < 0.001) and very strong correlation (r = 0.92, p < 0.001) between objective assessment tools was demonstrated. Skill estimation of OSATS and M-GEARS had a Spearman’s Correlation Coefficient 0.37 and 0.36, respectively, with normalised mean absolute error representing a prediction error of 17.92% (inverted “accuracy” 82.08%) and 20.6% (inverted “accuracy” 79.4%) respectively. The best performing models in error prediction achieved mean absolute precision of 37.14%, area under the curve 65.10% and Macro-F1 58.97%. // Conclusions: This is the first study to employ detailed error detection methodology and deep learning models within real robotic surgical video. This benchmark evaluation of AI models sets a foundation and promising approach for future advancements in automated technical skill assessment.
| Type: | Article |
|---|---|
| Title: | Deep learning prediction of error and skill in robotic prostatectomy suturing |
| Location: | Germany |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1007/s00464-024-11341-5 |
| Publisher version: | https://doi.org/10.1007/s00464-024-11341-5 |
| Language: | English |
| Additional information: | This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. |
| Keywords: | Robotic; Deep learning; Errors; Technical skill |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Medical Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Medical Sciences > Div of Surgery and Interventional Sci UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Med Phys and Biomedical Eng UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Medical Sciences > Div of Surgery and Interventional Sci > Department of Targeted Intervention |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10218341 |
Archive Staff Only
 |
View Item |
