TY  - GEN
AV  - public
SN  - 1050-4729
Y1  - 2024/08/08/
CY  - Yokohama, Japan
TI  - LLM-Assisted Multi-Teacher Continual Learning for Visual Question Answering in Robotic Surgery
ID  - discovery10197049
UR  - https://doi.org/10.1109/ICRA57147.2024.10610603
EP  - 10778
N2  - Visual question answering (VQA) can be fundamentally crucial for promoting robotic-assisted surgical education. In practice, the needs of trainees are constantly evolving, such as learning more surgical types and adapting to new surgical instruments/techniques. Therefore, continually updating the VQA system by a sequential data stream from multiple resources is demanded in robotic surgery to address new tasks. In surgical scenarios, the privacy issue of patient data often restricts the availability of old data when updating the model, necessitating an exemplar-free continual learning (CL) setup. However, prior studies overlooked two vital problems of the surgical domain: i) large domain shifts from diverse surgical operations collected from multiple departments or clinical centers, and ii) severe data imbalance arising from the uneven presence of surgical instruments or activities during surgical procedures. This paper proposes to address these two problems with a multimodal large language model (LLM) and an adaptive weight assignment methodology. We first develop a new multi-teacher CL framework that leverages a multimodal LLM as the additional teacher. The strong generalization ability of the LLM can bridge the knowledge gap when domain shifts and data imbalances occur. We then put forth a novel data processing method that transforms complex LLM embeddings into logits compatible with our CL framework. We also design an adaptive weight assignment approach that balances the generalization ability of the LLM and the domain expertise of the old CL model. Finally, we construct a new dataset for surgical VQA tasks. Extensive experimental results demonstrate the superiority of our method to other advanced CL models.
SP  - 10772
N1  - This version is the author accepted manuscript. For information on re-use, please refer to the publisher?s terms and conditions.
PB  - IEEE
A1  - Chen, K
A1  - Du, Y
A1  - You, T
A1  - Islam, M
A1  - Guo, Z
A1  - Jin, Y
A1  - Chen, G
A1  - Heng, PA
KW  - Continuing education
KW  -  
Adaptation models
KW  -  
Visualization
KW  -  
Instruments
KW  -  
Large language models
KW  -  
Surgery
KW  -  
Transforms
ER  -