Kim, Yunsoo;
Wu, Jinge;
Abdulle, Yusuf;
Gao, Yue;
Wu, Honghan;
(2024)
Enhancing Human-Computer Interaction in Chest X-Ray Analysis Using Vision and Language Model with Eye Gaze Patterns.
In: Linguraru, MG and Dou, Q and Feragen, A and Giannarou, S and Glocker, B and Lekadir, K and Schnabel, JA, (eds.)
Medical Image Computing and Computer Assisted Intervention – MICCAI 2024.
(pp. pp. 184-194).
Springer: Cham, Switzerland.
Preview |
Text
3261_paper.pdf - Accepted Version Download (274kB) | Preview |
Abstract
Recent advancements in Computer Assisted Diagnosis have shown promising performance in medical imaging tasks, particularly in chest X-ray analysis. However, the interaction between these models and radiologists has been primarily limited to input images. This work proposes a novel approach to enhance human-computer interaction in chest X-ray analysis using Vision-Language Models (VLMs) enhanced with radiologists’ attention by incorporating eye gaze data alongside textual prompts. Our approach leverages heatmaps generated from eye gaze data, overlaying them onto medical images to highlight areas of intense radiologist’s focus during chest X-ray evaluation. We evaluate this methodology in tasks such as visual question answering, chest X-ray report automation, error detection, and differential diagnosis. Our results demonstrate the inclusion of eye gaze information significantly enhances the accuracy of chest X-ray analysis. Also, the impact of eye gaze on fine-tuning was confirmed as it outperformed other medical VLMs in all tasks except visual question answering. This work marks the potential of leveraging both the VLM’s capabilities and the radiologist’s domain knowledge to improve the capabilities of AI models in medical imaging, paving a novel way for Computer Assisted Diagnosis with a humancentred AI. The code for processing data and evaluation can be found at https://github.com/knowlab/CXR_VLM_EyeGaze.
| Type: | Proceedings paper |
|---|---|
| Title: | Enhancing Human-Computer Interaction in Chest X-Ray Analysis Using Vision and Language Model with Eye Gaze Patterns |
| Event: | 27th International Conference: Medical Image Computing and Computer Assisted Intervention – MICCAI 2024 |
| Location: | MOROCCO, Palmeraie Conf Ctr, Marrakesh |
| Dates: | 6 Oct 2024 - 10 Oct 2024 |
| ISBN-13: | 978-3-031-72383-4 |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1007/978-3-031-72384-1_18 |
| Publisher version: | https://doi.org/10.1007/978-3-031-72384-1_18 |
| Language: | English |
| Additional information: | This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions. |
| Keywords: | Vision Language Model, Eye Gaze, Chest X-ray |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Population Health Sciences > Institute of Health Informatics UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Population Health Sciences > Institute of Health Informatics > Clinical Epidemiology |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10215971 |
Archive Staff Only
 |
View Item |
