UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

RecurBot: Learn to auto-complete GUI tasks from human demonstrations

Intharah, T; Firman, M; Brostow, GJ; (2018) RecurBot: Learn to auto-complete GUI tasks from human demonstrations. In: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. (pp. LBW595-LBW595). ACM: New York, NY, USA. Green open access

[img]
Preview
Text
RecurBot.pdf - Accepted version

Download (1MB) | Preview

Abstract

On the surface, task-completion should be easy in graphical user interface (GUI) settings. In practice however, different actions look alike and applications run in operating-system silos. Our aim within GUI action recognition and prediction is to help the user, at least in completing the tedious tasks that are largely repetitive. We propose a method that learns from a few user-performed demonstrations, and then predicts and finally performs the remaining actions in the task. For example, a user can send customized SMS messages to the first three contacts in a school’s spreadsheet of parents; then our system loops the process, iterating through the remaining parents. First, our analysis system segments the demonstration into discrete loops, where each iteration usually included both intentional and accidental variations. Our technical innovation approach is a solution to the standing motif-finding optimization problem, but we also find visual patterns in those intentional variations. The second challenge is to predict subsequent GUI actions, extrapolating the patterns to allow our system to predict and perform the rest of a task. We validate our approach on a new database of GUI tasks, and show that our system usually (a) gleans what it needs from short user demonstrations, and (b) auto-completes tasks in diverse GUI situations.

Type: Proceedings paper
Title: RecurBot: Learn to auto-complete GUI tasks from human demonstrations
Event: CHI EA '18
Open access status: An open access version is available from UCL Discovery
DOI: 10.1145/3170427.3188532
Publisher version: https://doi.org/10.1145/3170427.3188532
Language: English
Additional information: This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions.
UCL classification: UCL
UCL > Provost and Vice Provost Offices
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science
URI: https://discovery.ucl.ac.uk/id/eprint/10088947
Downloads since deposit
11Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item