UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

HILC: Domain-Independent PbD System Via Computer Vision and Follow-Up Questions

Intharah, T; Turmukhambetov, D; Brostow, GJ; (2019) HILC: Domain-Independent PbD System Via Computer Vision and Follow-Up Questions. ACM Transactions on Interactive Intelligent Systems , 9 (2-3) , Article 16. 10.1145/3234508. Green open access

Intharah_TiiS_final__HILC__Domain_Independent_PbD_system_via_Computer_Vision_and_Follow_up_Questions.pdf - Accepted version

Download (12MB) | Preview


Creating automation scripts for tasks involving Graphical User Interface (GUI) interactions is hard. It is challenging because not all software applications allow access to a program’s internal state, nor do they all have accessibility APIs. Although much of the internal state is exposed to the user through the GUI, it is hard to programmatically operate the GUI’s widgets. To that end, we developed a system prototype that learns by demonstration, called HILC (Help, It Looks Confusing). Users, both programmers and non-programmers, train HILC to synthesize a task script by demonstrating the task. A demonstration produces the needed screenshots and their corresponding mouse-keyboard signals. After the demonstration, the user answers follow-up questions. We propose a user-in-the-loop framework that learns to generate scripts of actions performed on visible elements of graphical applications. Although pure programming by demonstration is still unrealistic due to a computer’s limited understanding of user intentions, we use quantitative and qualitative experiments to show that non-programming users are willing and effective at answering follow-up queries posed by our system, to help with confusing parts of the demonstrations. Our models of events and appearances are surprisingly simple but are combined effectively to cope with varying amounts of supervision. The best available baseline, Sikuli Slides, struggled to assist users in the majority of the tests in our user study experiments. The prototype with our proposed approach successfully helped users accomplish simple linear tasks, complicated tasks (monitoring, looping, and mixed), and tasks that span across multiple applications. Even when both systems could ultimately perform a task, ours was trained and refined by the user in less time.

Type: Article
Title: HILC: Domain-Independent PbD System Via Computer Vision and Follow-Up Questions
Open access status: An open access version is available from UCL Discovery
DOI: 10.1145/3234508
Publisher version: https://doi.org/10.1145/3234508
Language: English
Additional information: This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions.
Keywords: Human-centered computing; Graphical user interfaces; User interface programming; Computing methodologies; Activity recognition and understanding; Software and its engineering; Programming by example; Programming by Demonstration; GUI Automation; Action Segmentation and Recognition; Visual-based Programming by Demonstration
UCL classification: UCL
UCL > Provost and Vice Provost Offices
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science
URI: https://discovery.ucl.ac.uk/id/eprint/10075771
Downloads since deposit
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item