Intharah, T;
Firman, M;
Brostow, GJ;
(2018)
RecurBot: Learn to auto-complete GUI tasks from human demonstrations.
In:
Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems.
(pp. LBW595-LBW595).
ACM: New York, NY, USA.
Preview |
Text
RecurBot.pdf - Accepted Version Download (1MB) | Preview |
Abstract
On the surface, task-completion should be easy in graphical user interface (GUI) settings. In practice however, different actions look alike and applications run in operating-system silos. Our aim within GUI action recognition and prediction is to help the user, at least in completing the tedious tasks that are largely repetitive. We propose a method that learns from a few user-performed demonstrations, and then predicts and finally performs the remaining actions in the task. For example, a user can send customized SMS messages to the first three contacts in a school’s spreadsheet of parents; then our system loops the process, iterating through the remaining parents. First, our analysis system segments the demonstration into discrete loops, where each iteration usually included both intentional and accidental variations. Our technical innovation approach is a solution to the standing motif-finding optimization problem, but we also find visual patterns in those intentional variations. The second challenge is to predict subsequent GUI actions, extrapolating the patterns to allow our system to predict and perform the rest of a task. We validate our approach on a new database of GUI tasks, and show that our system usually (a) gleans what it needs from short user demonstrations, and (b) auto-completes tasks in diverse GUI situations.
| Type: | Proceedings paper |
|---|---|
| Title: | RecurBot: Learn to auto-complete GUI tasks from human demonstrations |
| Event: | CHI EA '18 |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1145/3170427.3188532 |
| Publisher version: | https://doi.org/10.1145/3170427.3188532 |
| Language: | English |
| Additional information: | This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10088947 |
Archive Staff Only
 |
View Item |
