Giunchi, Daniele;
Numan, Nels;
Gatti, Elia;
Steed, Anthony;
(2024)
DreamCodeVR: Towards Democratizing Behavior Design in Virtual Reality with Speech-Driven Programming.
In:
Proceedings of the 2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR).
IEEE (Institute of Electrical and Electronics Engineers): Orlando, FL, USA.
Preview |
Text
2024019754.pdf - Accepted Version Download (1MB) | Preview |
Abstract
Virtual Reality (VR) has revolutionized how we interact with digital worlds. However, programming for VR remains a complex and challenging task, requiring specialized skills and knowledge. Powered by large language models (LLMs), DreamCodeVR is designed to assist users, irrespective of their coding skills, in crafting basic object behavior in VR environments by translating spoken language into code within an active application. This approach seeks to simplify the process of defining behaviors visual changes through speech. Our preliminary user study indicated that the system’s speech interface supports elementary programming tasks, highlighting its potential to improve accessibility for users with varying technical skills. However, it also uncovered a wide range of challenges and opportunities. In an extensive discussion, we detail the system’s strengths, weaknesses, and areas for future research.
| Type: | Proceedings paper |
|---|---|
| Title: | DreamCodeVR: Towards Democratizing Behavior Design in Virtual Reality with Speech-Driven Programming |
| Event: | 2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR) |
| Location: | Orlando, Florida |
| Dates: | 16 Mar 2024 - 21 Apr 2024 |
| ISBN-13: | 979-8-3503-7402-5 |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1109/VR58804.2024.00078 |
| Publisher version: | https://doi.org/10.1109/VR58804.2024.00078 |
| Language: | English |
| Additional information: | This version is the author-accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions. |
| Keywords: | large language model, VR, rapid prototyping, low code programming, speech programming, Human-centered computing, Virtual reality Human-centered computing, Natural language interfaces |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10190408 |
Archive Staff Only
 |
View Item |
