[Paper Review] Learning Temporal Transformations From Time-Lapse Videos

Yipin Zhou, Tamara L. Berg|arXiv (Cornell University)|Aug 27, 2016

Generative Adversarial Networks and Image Synthesis29 references4 citations

TL;DR

This paper proposes deep learning models to predict long-term temporal transformations in time-lapse videos, using autoencoders, GANs, and recurrent networks to generate future object states. It introduces a novel dataset of 1,463 time-lapse videos across four transformations (melting, blooming, baking, rotting), demonstrating that adversarial training and fine-tuning significantly improve generation quality and human preference over baselines.

ABSTRACT

Based on life-long observations of physical, chemical, and biologic phenomena in the natural world, humans can often easily picture in their minds what an object will look like in the future. But, what about computers? In this paper, we learn computational models of object transformations from time-lapse videos. In particular, we explore the use of generative models to create depictions of objects at future times. These models explore several different prediction tasks: generating a future state given a single depiction of an object, generating a future state given two depictions of an object at different times, and generating future states recursively in a recurrent framework. We provide both qualitative and quantitative evaluations of the generated results, and also conduct a human evaluation to compare variations of our models.

Motivation & Objective

• To develop computational models that predict long-term object transformations from visual observation, inspired by human intuitive understanding of physical change.
• To address the challenge of modeling natural, continuous transformations (e.g., melting, rotting) over extended time scales (minutes to days) using video data.
• To evaluate model performance through quantitative metrics, human preference studies, and visualizations of learned motion patterns.
• To explore the effectiveness of adversarial training and pre-training/fine-tuning in improving generation quality for future state prediction.

Proposed method

• Trained autoencoder-based architectures to generate future object states from a single input image, with conditional labeling for time step prediction.
• Introduced a two-stack model that uses two input frames separated by a variable time gap to predict the state at double that interval.
• Employed a recurrent neural network to recursively generate multiple future frames from a single initial image, enabling long-horizon prediction.
• Applied generative adversarial networks (GANs) to improve realism of generated images by training a discriminator network alongside the generator.
• Used pre-training on a large reconstruction dataset followed by fine-tuning on time-lapse data to improve generalization and performance.
• Visualized learned transformation patterns using optical flow maps computed between input and generated images, clustering flows to reveal spatial trends.

Experimental results

Research questions

• RQ1 Can deep generative models accurately predict future object states from time-lapse videos across diverse physical transformations?
• RQ2 How do adversarial training and pre-training/fine-tuning improve the realism and accuracy of generated future states compared to standard autoencoders?
• RQ3 To what extent do human observers prefer model-generated future states, and how do different architectures compare in human evaluation?
• RQ4 What underlying motion patterns and spatial trends do the models learn, and can these be visualized meaningfully through optical flow analysis?

Key findings

• The p_g_mse+adv+ft method (generative adversarial network with pre-training and fine-tuning) achieved the highest human preference rate, with 38.2% of selections in pairwise generation, significantly outperforming the baseline (13.2%).
• In two-stack generation, the p_g_mse+adv+ft method achieved a 43.2% human preference rate for rotting, the highest among all methods.
• For recurrent generation, the p_g_mse+adv+ft method achieved 38.2% human preference, demonstrating strong performance in long-horizon prediction.
• Image retrieval accuracy improved with fine-tuning: top-1 accuracy reached 90% for p_g_mse+adv+ft, compared to 68% for p_mse+adv.
• Optical flow visualizations revealed consistent spatial trends: blooming showed outward growth (x-axis flow), melting showed downward shrinkage (y-axis flow), baking showed vertical inflation, and rotting showed upper-body inflation or dehydration.
• The average reconstruction error across all transformations was reduced to 0.3815 for p_g_mse+adv+ft in recurrent generation, indicating improved model fidelity.