[Paper Review] Image2GIF: Generating Cinemagraphs using Recurrent Deep Q-Networks

Yipin Zhou, Yale Song+1 more|arXiv (Cornell University)|Jan 27, 2018

Generative Adversarial Networks and Image Synthesis34 references3 citations

TL;DR

This paper proposes a recurrent generative model enhanced with Deep Q-Networks (DQN) to automatically generate cinemagraphs from a single static image, learning both where to animate (localization) and how to animate (motion generation). The method achieves superior visual quality and realism over baselines, with human evaluations showing RNN+DQN outperforms other methods in perceived realism and preference on both synthetic and real datasets.

ABSTRACT

Given a still photograph, one can imagine how dynamic objects might move against a static background. This idea has been actualized in the form of cinemagraphs, where the motion of particular objects within a still image is repeated, giving the viewer a sense of animation. In this paper, we learn computational models that can generate cinemagraph sequences automatically given a single image. To generate cinemagraphs, we explore combining generative models with a recurrent neural network and deep Q-networks to enhance the power of sequence generation. To enable and evaluate these models we make use of two datasets, one synthetically generated and the other containing real video generated cinemagraphs. Both qualitative and quantitative evaluations demonstrate the effectiveness of our models on the synthetic and real datasets.

Motivation & Objective

• To automate the creation of cinemagraphs from a single static image, reducing manual effort in generating animated GIFs.
• To learn which objects in a scene can move and how they should move, capturing dynamic semantics from static input.
• To develop a generative model that combines temporal modeling with reinforcement learning for improved motion sequence generation.
• To evaluate the model on both synthetic and real-world cinemagraph datasets, enabling robust comparison.

Proposed method

• The model uses a recurrent neural network (RNN) with a variational autoencoder to encode and generate sequential frames from a single input image.
• A Deep Q-Network (DQN) is integrated to guide stochastic decision-making in selecting which image regions to animate and how to animate them over time.
• The generative model is trained using a combination of adversarial loss and reconstruction loss to enhance visual realism and structural consistency.
• A recurrent architecture enables the model to generate temporally coherent sequences by conditioning each frame on prior generated frames.
• The model is trained and evaluated on two datasets: a synthetic dataset with controlled motion and a real-world dataset of Internet-captured cinemagraphs.
• Segmentation maps are used to evaluate motion localization accuracy, comparing predicted moving object centers with ground truth.

Experimental results

Research questions

• RQ1 Can a deep learning model automatically generate temporally coherent cinemagraph sequences from a single static image?
• RQ2 How well can a model localize moving objects within a static image and generate plausible motion sequences for them?
• RQ3 To what extent does integrating Deep Q-Networks improve the quality and realism of generated cinemagraphs compared to baseline methods?
• RQ4 How does the model perform on both synthetic data with ground-truth motion and real-world cinemagraphs?
• RQ5 Can human evaluators distinguish generated cinemagraphs from real ones, and how does the model compare to prior work in perceived realism?

Key findings

• The RNN+DQN model achieved the highest human preference in both qualitative comparisons, with 63.3% of subjects selecting it as the best cinemagraph in pairwise comparisons.
• The model was able to fool human observers into believing generated cinemagraphs were real 60.0% of the time, significantly outperforming the baseline method.
• On the synthetic dataset, the RNN+DQN model reduced the average Euclidean distance between predicted and ground-truth object centers by 28.7% compared to baseline, indicating better motion localization.
• The ablation study showed that using small bottleneck (z) and Q-vector sizes severely degraded performance due to information loss and over-quantization.
• The constant baseline (copying the first frame) achieved zero PSNR/SSIM on frame-wise difference images, while RNN+DQN achieved non-zero scores, confirming its ability to generate dynamic motion.
• The model demonstrated robustness to hyperparameter choices, with optimal performance at 64-dimensional z and 512-dimensional Q-vector on synthetic and real data, respectively.