UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

Help, It Looks Confusing: GUI Task Automation Through Demonstration and Follow-up Questions

Intharah, T; Turmukhambetov, D; Brostow, GJ; (2017) Help, It Looks Confusing: GUI Task Automation Through Demonstration and Follow-up Questions. In: Proceedings of the 22nd International Conference on Intelligent User Interfaces. (pp. pp. 233-243). ACM: New York, NY, USA. Green open access

[img]
Preview
Text
iui17_preprint_HILC.pdf - Accepted version

Download (5MB) | Preview

Abstract

Non-programming users should be able to create their own customized scripts to perform computer-based tasks for them, just by demonstrating to the machine how it's done. To that end, we develop a system prototype which learns-by-demonstration called HILC (Help, It Looks Confusing). Users train HILC to synthesize a task script by demonstrating the task, which produces the needed screenshots and their corresponding mouse-keyboard signals. After the demonstration, the user answers follow-up questions. We propose a user-in-the-loop framework that learns to generate scripts of actions performed on visible elements of graphical applications. While pure programming-by-demonstration is still unrealistic, we use quantitative and qualitative experiments to show that non-programming users are willing and effective at answering follow-up queries posed by our system. Our models of events and appearance are surprisingly simple, but are combined effectively to cope with varying amounts of supervision. The best available baseline, Sikuli Slides, struggled with the majority of the tests in our user study experiments. The prototype with our proposed approach successfully helped users accomplish simple linear tasks, complicated tasks (monitoring, looping, and mixed), and tasks that span across multiple executables. Even when both systems could ultimately perform a task, ours was trained and refined by the user in less time.

Type: Proceedings paper
Title: Help, It Looks Confusing: GUI Task Automation Through Demonstration and Follow-up Questions
Event: 22nd International Conference on Intelligent User Interfaces (IUI '17)
Location: Limassol, CYPRUS
Dates: 13 March 2017 - 16 March 2017
Open access status: An open access version is available from UCL Discovery
DOI: 10.1145/3025171.3025176
Publisher version: https://doi.org/10.1145/3025171.3025176
Language: English
Additional information: This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions.
Keywords: Science & Technology, Technology, Computer Science, Artificial Intelligence, Computer Science, Cybernetics, Computer Science, Theory & Methods, Computer Science, Programming by Demonstration, GUI Automation, Action Segmentation and Recognition
UCL classification: UCL
UCL > Provost and Vice Provost Offices
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science
URI: https://discovery.ucl.ac.uk/id/eprint/10088945
Downloads since deposit
2Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item