eprintid: 10205008 rev_number: 7 eprint_status: archive userid: 699 dir: disk0/10/20/50/08 datestamp: 2025-02-20 11:38:58 lastmod: 2025-02-20 11:38:58 status_changed: 2025-02-20 11:38:58 type: article metadata_visibility: show sword_depositor: 699 creators_name: Thieme, Anja creators_name: Rajamohan, Abhijith creators_name: Cooper, Benjamin creators_name: Groombridge, Heather creators_name: Simister, Robert creators_name: Wong, Barney creators_name: Woznitza, Nicholas creators_name: Pinnock, Mark A creators_name: Wetscherek, Maria T creators_name: Morrison, Cecily creators_name: Richardson, Hannah creators_name: Pérez-García, Fernando creators_name: Hyland, Stephanie L creators_name: Bannur, Shruthi creators_name: Castro, Daniel C creators_name: Bouzid, Kenza creators_name: Schwaighofer, Anton creators_name: Ranjit, Mercy P creators_name: Sharma, Harshita creators_name: Lungren, Matthew P creators_name: Oktay, Ozan creators_name: Alvarez-Valle, Javier creators_name: Nori, Aditya creators_name: Harris, Stephen creators_name: Jacob, Joseph title: Challenges for Responsible AI Design and Workflow Integration in Healthcare: A Case Study of Automatic Feeding Tube Qualification in Radiology ispublished: inpress divisions: UCL divisions: B02 divisions: B04 divisions: C10 divisions: D17 divisions: DD4 divisions: F48 divisions: K71 divisions: J73 keywords: Radiology, AI, healthcare, responsible AI, socio-technical systems, feeding tubes, NGT note: This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions. abstract: Nasogastric tubes (NGTs) are feeding tubes that are inserted through the nose into the stomach to deliver nutrition or medication. If not placed correctly, they can cause serious harm, even death to patients. Recent AI developments demonstrate the feasibility of robustly detecting NGT placement from Chest X-ray images to reduce risks of sub-optimally or critically placed NGTs being missed or delayed in their detection, but gaps remain in clinical practice integration. In this study, we present a human-centered approach to the problem and describe insights derived following contextual inquiry and in-depth interviews with 15 clinical stakeholders. The interviews helped understand challenges in existing workflows, and how best to align technical capabilities with user needs and expectations. We discovered the trade-offs and complexities that need consideration when choosing suitable workflow stages, target users, and design configurations for different AI proposals. We explored how to balance AI benefits and risks for healthcare staff and patients within broader organizational, technical, and medical-legal constraints. We also identified data issues related to edge cases and data biases that affect model training and evaluation; how data documentation practices influence data preparation and labelling; and how to measure relevant AI outcomes reliably in future evaluations. We discuss how our work informs design and development of AI applications that are clinically useful, ethical, and acceptable in real-world healthcare services. date: 2025-02-12 date_type: published publisher: Association for Computing Machinery (ACM) official_url: https://doi.org/10.1145/3716500 oa_status: green full_text_type: other language: eng primo: open primo_central: open_green verified: verified_manual elements_id: 2362138 doi: 10.1145/3716500 lyricists_name: Pinnock, Mark lyricists_name: Harris, Stephen lyricists_name: Jacob, Joseph lyricists_id: MAPIN75 lyricists_id: HARRI92 lyricists_id: JJACO76 actors_name: Flynn, Bernadette actors_id: BFFLY94 actors_role: owner full_text_status: public publication: ACM Transactions on Computer-Human Interaction issn: 1073-0516 citation: Thieme, Anja; Rajamohan, Abhijith; Cooper, Benjamin; Groombridge, Heather; Simister, Robert; Wong, Barney; Woznitza, Nicholas; ... Jacob, Joseph; + view all <#> Thieme, Anja; Rajamohan, Abhijith; Cooper, Benjamin; Groombridge, Heather; Simister, Robert; Wong, Barney; Woznitza, Nicholas; Pinnock, Mark A; Wetscherek, Maria T; Morrison, Cecily; Richardson, Hannah; Pérez-García, Fernando; Hyland, Stephanie L; Bannur, Shruthi; Castro, Daniel C; Bouzid, Kenza; Schwaighofer, Anton; Ranjit, Mercy P; Sharma, Harshita; Lungren, Matthew P; Oktay, Ozan; Alvarez-Valle, Javier; Nori, Aditya; Harris, Stephen; Jacob, Joseph; - view fewer <#> (2025) Challenges for Responsible AI Design and Workflow Integration in Healthcare: A Case Study of Automatic Feeding Tube Qualification in Radiology. ACM Transactions on Computer-Human Interaction 10.1145/3716500 <https://doi.org/10.1145/3716500>. (In press). Green open access document_url: https://discovery.ucl.ac.uk/id/eprint/10205008/1/3716500.pdf