Intharah, T;
Turmukhambetov, D;
Brostow, GJ;
(2019)
HILC: Domain-Independent PbD System Via Computer Vision and Follow-Up Questions.
ACM Transactions on Interactive Intelligent Systems
, 9
(2-3)
, Article 16. 10.1145/3234508.
Preview |
Text
Intharah_TiiS_final__HILC__Domain_Independent_PbD_system_via_Computer_Vision_and_Follow_up_Questions.pdf - Accepted Version Download (12MB) | Preview |
Abstract
Creating automation scripts for tasks involving Graphical User Interface (GUI) interactions is hard. It is challenging because not all software applications allow access to a program’s internal state, nor do they all have accessibility APIs. Although much of the internal state is exposed to the user through the GUI, it is hard to programmatically operate the GUI’s widgets. To that end, we developed a system prototype that learns by demonstration, called HILC (Help, It Looks Confusing). Users, both programmers and non-programmers, train HILC to synthesize a task script by demonstrating the task. A demonstration produces the needed screenshots and their corresponding mouse-keyboard signals. After the demonstration, the user answers follow-up questions. We propose a user-in-the-loop framework that learns to generate scripts of actions performed on visible elements of graphical applications. Although pure programming by demonstration is still unrealistic due to a computer’s limited understanding of user intentions, we use quantitative and qualitative experiments to show that non-programming users are willing and effective at answering follow-up queries posed by our system, to help with confusing parts of the demonstrations. Our models of events and appearances are surprisingly simple but are combined effectively to cope with varying amounts of supervision. The best available baseline, Sikuli Slides, struggled to assist users in the majority of the tests in our user study experiments. The prototype with our proposed approach successfully helped users accomplish simple linear tasks, complicated tasks (monitoring, looping, and mixed), and tasks that span across multiple applications. Even when both systems could ultimately perform a task, ours was trained and refined by the user in less time.
| Type: | Article |
|---|---|
| Title: | HILC: Domain-Independent PbD System Via Computer Vision and Follow-Up Questions |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1145/3234508 |
| Publisher version: | https://doi.org/10.1145/3234508 |
| Language: | English |
| Additional information: | This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions. |
| Keywords: | Human-centered computing; Graphical user interfaces; User interface programming; Computing methodologies; Activity recognition and understanding; Software and its engineering; Programming by example; Programming by Demonstration; GUI Automation; Action Segmentation and Recognition; Visual-based Programming by Demonstration |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10075771 |
Archive Staff Only
 |
View Item |
