Tang, Kaiyuan;
Quinlan, Mark;
Cho, Youngjun;
(2025)
Exploring LLM Agents as Seamless and Context-aware UI Adjusters for the Visually Impaired.
In:
Proceedings of the 2025 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp Companion ’25).
ACM: Espoo, Finland.
(In press).
Preview |
PDF
camera-ready.pdf - Accepted Version Download (2MB) | Preview |
Abstract
The user interface (UI) of traditional electronic devices often lack intuitive and flexible methods to accommodate the diverse needs of visually impaired (VI) users. This paper explores the role of Large Language Model (LLM)-powered Conversational Agents (CAs) serving as intelligent, context-aware mediators to achieve VI-friendly adaptive UI. Our approach focuses on facilitating real-time UI adjustments based on human-AI conversations and device and ambient context, leveraging LLM CAs’ reasoning and tool-using capabilities to accommodate different types of visual impairments. A prototype was developed to demonstrate how the system can dynamically adapt the interface. Our initial observations suggest that this approach has the potential to promote a seamless and inclusive experience for VI users, particularly on daily information terminals that are not flexible or VI-friendly in terms of quickly adjusting UI settings, such as kiosks, multimedia guides, and wearable displays.
| Type: | Proceedings paper |
|---|---|
| Title: | Exploring LLM Agents as Seamless and Context-aware UI Adjusters for the Visually Impaired |
| Event: | the 2025 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp Companion ’25) |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1145/3714394.3754391 |
| Publisher version: | https://doi.org/10.1145/3714394.3754391 |
| Language: | English |
| Additional information: | © 2025 Copyright held by the owner/author(s). This work is licensed under a Creative Commons Attribution 4.0 International License. |
| Keywords: | Assistive Technology; Adaptive UI; Visually Impaired Assistance; Context-Aware Computing; Conversational Interactions; Large Language Models; Multi-agent Systems; Ambient Computing |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10216164 |
Archive Staff Only
 |
View Item |
