Zu, Weiqin; Song, Wenbin; Chen, Ruiqing; Guo, Ze; Sun, Fanglei; Tian, Zheng; Pan, Wei; Zu, Weiqin; Song, Wenbin; Chen, Ruiqing; Guo, Ze; Sun, Fanglei; Tian, Zheng; Pan, Wei; Wang, Jun; - view fewer (2024) Language and Sketching: An LLM-driven Interactive Multimodal Multitask Robot Navigation Framework. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA) 2024. (pp. pp. 1019-1025). IEEE

Preview

Text 2311.08244v2.pdf - Accepted Version Download (34MB) | Preview

Abstract

The socially-aware navigation system has evolved to adeptly avoid various obstacles while performing multiple tasks, such as point-to-point navigation, human-following, and-guiding. However, a prominent gap persists: in Human-Robot Interaction (HRI), the procedure of communicating commands to robots demands intricate mathematical formulations. Furthermore, the transition between tasks does not quite possess the intuitive control and user-centric interactivity that one would desire. In this work, we propose an LLM-driven interactive multimodal multitask robot navigation framework, termed LIM2N, to solve the above new challenge in the navigation field. We achieve this by first introducing a multimodal interaction framework where language and hand-drawn inputs can serve as navigation constraints and control objectives. Next, a reinforcement learning agent is built to handle multiple tasks with the received information. Crucially, LIM2N creates smooth cooperation among the reasoning of multimodal input, multitask planning, and adaptation and processing of the intelligent sensing modules in the complicated system. Detailed experiments are conducted in both simulation and the real world demonstrating that LIM2N has solid user needs understanding, alongside an enhanced interactive experience.

Download activity - last month

Export as JSONXMLCSV

Download activity - last 12 months

Export as XMLCSVJSON

Downloads by country - last 12 months

Export as JSONCSVXML

Archive Staff Only

View Item