UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

DreamCodeVR: Towards Democratizing Behavior Design in Virtual Reality with Speech-Driven Programming

Giunchi, Daniele; Numan, Nels; Gatti, Elia; Steed, Anthony; (2024) DreamCodeVR: Towards Democratizing Behavior Design in Virtual Reality with Speech-Driven Programming. In: Proceedings of the 2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR). IEEE (Institute of Electrical and Electronics Engineers): Orlando, FL, USA. Green open access

[thumbnail of 2024019754.pdf]
Preview
Text
2024019754.pdf - Accepted Version

Download (1MB) | Preview

Abstract

Virtual Reality (VR) has revolutionized how we interact with digital worlds. However, programming for VR remains a complex and challenging task, requiring specialized skills and knowledge. Powered by large language models (LLMs), DreamCodeVR is designed to assist users, irrespective of their coding skills, in crafting basic object behavior in VR environments by translating spoken language into code within an active application. This approach seeks to simplify the process of defining behaviors visual changes through speech. Our preliminary user study indicated that the system’s speech interface supports elementary programming tasks, highlighting its potential to improve accessibility for users with varying technical skills. However, it also uncovered a wide range of challenges and opportunities. In an extensive discussion, we detail the system’s strengths, weaknesses, and areas for future research.

Type: Proceedings paper
Title: DreamCodeVR: Towards Democratizing Behavior Design in Virtual Reality with Speech-Driven Programming
Event: 2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR)
Location: Orlando, Florida
Dates: 16 Mar 2024 - 21 Apr 2024
ISBN-13: 979-8-3503-7402-5
Open access status: An open access version is available from UCL Discovery
DOI: 10.1109/VR58804.2024.00078
Publisher version: https://doi.org/10.1109/VR58804.2024.00078
Language: English
Additional information: This version is the author-accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions.
Keywords: large language model, VR, rapid prototyping, low code programming, speech programming, Human-centered computing, Virtual reality Human-centered computing, Natural language interfaces
UCL classification: UCL
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science
URI: https://discovery.ucl.ac.uk/id/eprint/10190408
Downloads since deposit
66Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item