UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

Single Image Video Prediction with Auto-Regressive GANs

Huang, J; Chia, YK; Yu, S; Yee, K; Küster, D; Krumhuber, EG; Herremans, D; (2022) Single Image Video Prediction with Auto-Regressive GANs. Sensors , 22 (9) , Article 3533. 10.3390/s22093533. Green open access

[thumbnail of sensors-22-03533-v2.pdf]
Preview
Text
sensors-22-03533-v2.pdf - Published Version

Download (106MB) | Preview

Abstract

In this paper, we introduce an approach for future frames prediction based on a single input image. Our method is able to generate an entire video sequence based on the information contained in the input frame. We adopt an autoregressive approach in our generation process, i.e., the output from each time step is fed as the input to the next step. Unlike other video prediction methods that use “one shot” generation, our method is able to preserve much more details from the input image, while also capturing the critical pixel-level changes between the frames. We overcome the problem of generation quality degradation by introducing a “complementary mask” module in our architecture, and we show that this allows the model to only focus on the generation of the pixels that need to be changed, and to reuse those that should remain static from its previous frame. We empirically validate our methods against various video prediction models on the UT Dallas Dataset, and show that our approach is able to generate high quality realistic video sequences from one static input image. In addition, we also validate the robustness of our method by testing a pre-trained model on the unseen ADFES facial expression dataset. We also provide qualitative results of our model tested on a human action dataset: The Weizmann Action database.

Type: Article
Title: Single Image Video Prediction with Auto-Regressive GANs
Location: Switzerland
Open access status: An open access version is available from UCL Discovery
DOI: 10.3390/s22093533
Publisher version: https://doi.org/10.3390/s22093533
Language: English
Additional information: This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited
Keywords: Autoregressive GANs, emotion generation, video prediction
UCL classification: UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences
UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > Div of Psychology and Lang Sciences > Experimental Psychology
UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences
UCL
UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > Div of Psychology and Lang Sciences
URI: https://discovery.ucl.ac.uk/id/eprint/10149182
Downloads since deposit
8Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item