Single Image Video Prediction with Auto-Regressive GANs

  • In this paper, we introduce an approach for future frames prediction based on a single input image. Our method is able to generate an entire video sequence based on the information contained in the input frame. We adopt an autoregressive approach in our generation process, i.e., the output from each time step is fed as the input to the next step. Unlike other video prediction methods that use “one shot” generation, our method is able to preserve much more details from the input image, while also capturing the critical pixel-level changes between the frames. We overcome the problem of generation quality degradation by introducing a “complementary mask” module in our architecture, and we show that this allows the model to only focus on the generation of the pixels that need to be changed, and to reuse those that should remain static from its previous frame. We empirically validate our methods against various video prediction models on the UT Dallas Dataset, and show that our approach is able to generate high quality realistic video sequences from one static input image. In addition, we also validate the robustness of our method by testing a pre-trained model on the unseen ADFES facial expression dataset. We also provide qualitative results of our model tested on a human action dataset: The Weizmann Action database.

Download full text files

Export metadata

Metadaten
Author:Jiahui HuangORCiD, Yew Ken ChiaORCiD, Samson YuORCiD, Kevin Yee, Dennis KüsterORCiDGND, Eva G. KrumhuberORCiD, Dorien HerremansORCiDGND, Gemma Roig NogueraORCiDGND
URN:urn:nbn:de:hebis:30:3-718955
DOI:https://doi.org/10.3390/s22093533
ISSN:1424-8220
Parent Title (English):Sensors
Publisher:MDPI
Place of publication:Basel
Document Type:Article
Language:English
Date of Publication (online):2022/05/06
Date of first Publication:2022/05/06
Publishing Institution:Universitätsbibliothek Johann Christian Senckenberg
Release Date:2023/02/08
Tag:autoregressive GANs; emotion generation; video prediction
Volume:22
Issue:9, art. 3533
Article Number:3533
Page Number:14
First Page:1
Last Page:14
Note:
This project was partially supported by the Singapore Ministry of Education, Grant no. MOE2018-T2-2-161.
HeBIS-PPN:507187016
Institutes:Informatik und Mathematik
Dewey Decimal Classification:0 Informatik, Informationswissenschaft, allgemeine Werke / 00 Informatik, Wissen, Systeme / 004 Datenverarbeitung; Informatik
1 Philosophie und Psychologie / 15 Psychologie / 150 Psychologie
6 Technik, Medizin, angewandte Wissenschaften / 62 Ingenieurwissenschaften / 620 Ingenieurwissenschaften und zugeordnete Tätigkeiten
Sammlungen:Universitätspublikationen
Licence (German):License LogoCreative Commons - CC BY - Namensnennung 4.0 International