[Paper Review] Animating Landscape: Self-Supervised Learning of Decoupled Motion and Appearance for Single-Image Video Synthesis

Yuki Endo, Yoshihiro Kanamori+1 more|arXiv (Cornell University)|Oct 16, 2019

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes a self-supervised, decoupled deep learning method for generating high-resolution, long-term animations from a single landscape image by separately predicting motion (via backward flow fields) and appearance (via color transfer functions). By applying these intermediate representations only to the original input image per frame, the approach avoids error accumulation and enables controllable, cyclic video synthesis with superior temporal coherence and resolution compared to prior methods.

ABSTRACT

Automatic generation of a high-quality video from a single image remains a challenging task despite the recent advances in deep generative models. This paper proposes a method that can create a high-resolution, long-term animation using convolutional neural networks (CNNs) from a single landscape image where we mainly focus on skies and waters. Our key observation is that the motion (e.g., moving clouds) and appearance (e.g., time-varying colors in the sky) in natural scenes have different time scales. We thus learn them separately and predict them with decoupled control while handling future uncertainty in both predictions by introducing latent codes. Unlike previous methods that infer output frames directly, our CNNs predict spatially-smooth intermediate data, i.e., for motion, flow fields for warping, and for appearance, color transfer maps, via self-supervised learning, i.e., without explicitly-provided ground truth. These intermediate data are applied not to each previous output frame, but to the input image only once for each output frame. This design is crucial to alleviate error accumulation in long-term predictions, which is the essential problem in previous recurrent approaches. The output frames can be looped like cinemagraph, and also be controlled directly by specifying latent codes or indirectly via visual annotations. We demonstrate the effectiveness of our method through comparisons with the state-of-the-arts on video prediction as well as appearance manipulation.

Motivation & Objective

• To address the challenge of generating high-quality, long-term videos from a single static landscape image, where motion and appearance vary on different time scales.
• To reduce error accumulation in long-term video prediction by decoupling motion and appearance prediction and applying them directly to the input image rather than recurrently to generated frames.
• To enable controllable video generation through latent codes and visual annotations, offering both direct and indirect control over motion and appearance.
• To improve video quality and resolution beyond state-of-the-art methods by leveraging self-supervised learning on time-lapse video datasets without ground-truth supervision.

Proposed method

• The method uses two separate convolutional neural networks: one for predicting backward optical flow fields to model motion, and another for predicting spatially smooth color transfer functions to model appearance changes.
• Motion is modeled as frame-to-frame differences using backward flow fields, which are applied to warp the input image to generate intermediate motion-affected frames.
• Appearance variation is modeled as color transfer functions between the input frame and arbitrary frames in time-lapse videos, which are then temporally interpolated for smooth color transitions.
• Latent codes are used to model uncertainty in future predictions, with regularization to ensure plausible and diverse outputs during inference.
• The final video frames are synthesized by applying the predicted flow fields and color transfer functions to the original input image only once per output frame, avoiding recurrent error propagation.
• Control is enabled via latent code sampling from a learned codebook or via visual annotations, allowing direct or indirect manipulation of motion and appearance.

Experimental results

Research questions

• RQ1 Can motion and appearance in natural landscapes be effectively decoupled and modeled with different time scales to improve video synthesis quality?
• RQ2 Can self-supervised learning on time-lapse videos enable high-resolution, long-term video generation without ground-truth supervision?
• RQ3 Does applying intermediate representations (flow fields and color transfer functions) directly to the input image reduce error accumulation compared to recurrent generation?
• RQ4 Can latent codes enable controllable, diverse, and plausible video generation from a single image?
• RQ5 How does the decoupled, intermediate representation approach compare to end-to-end frame prediction in terms of resolution, coherence, and temporal stability?

Key findings

• The method generates high-resolution videos at 1024×576 resolution, significantly exceeding the resolution of prior single-image video synthesis methods.
• The approach achieves longer-term video sequences with reduced error accumulation, as evidenced by stable visual quality even at 500 frames, though very long sequences show some distortions.
• Motion and appearance can be independently controlled via latent codes or visual annotations, enabling diverse and plausible animations from the same input image.
• The use of intermediate representations (flow fields and color transfer functions) applied only to the input image prevents error accumulation common in recurrent generation, improving long-term coherence.
• Quantitative and qualitative comparisons show superior performance over state-of-the-art video prediction and appearance manipulation models in both visual quality and temporal consistency.
• The method successfully generates cyclic animations that loop naturally, demonstrating its ability to model realistic, time-varying scene dynamics.