[Paper Review] Decomposing Motion and Content for Natural Video Sequence Prediction

Ruben Villegas, Shuicheng Yan+3 more|arXiv (Cornell University)|Jun 25, 2017

Video Analysis and Summarization10 references416 citations

TL;DR

MCnet decomposes video prediction into separate motion and content encoders, enabling end-to-end training for pixel-level future frame prediction and achieving state-of-the-art results on multiple action video datasets.

ABSTRACT

We propose a deep neural network for the prediction of future frames in natural video sequences. To effectively handle complex evolution of pixels in videos, we propose to decompose the motion and content, two key components generating dynamics in videos. Our model is built upon the Encoder-Decoder Convolutional Neural Network and Convolutional LSTM for pixel-level prediction, which independently capture the spatial layout of an image and the corresponding temporal dynamics. By independently modeling motion and content, predicting the next frame reduces to converting the extracted content features into the next frame content by the identified motion features, which simplifies the task of prediction. Our model is end-to-end trainable over multiple time steps, and naturally learns to decompose motion and content without separate training. We evaluate the proposed network architecture on human activity videos using KTH, Weizmann action, and UCF-101 datasets. We show state-of-the-art performance in comparison to recent approaches. To the best of our knowledge, this is the first end-to-end trainable network architecture with motion and content separation to model the spatiotemporal dynamics for pixel-level future prediction in natural videos.

Motivation & Objective

• Motivate and address pixel-level future frame prediction in natural videos.
• Propose a two-stream Architecture that separately encodes motion and content.
• Show that end-to-end training can learn motion-content decomposition without supervision.

Proposed method

• Two encoder pathways: a motion encoder processing frame differences with a ConvLSTM to capture dynamics, and a content encoder processing the last observed frame to capture spatial layout.
• Multi-scale motion-content residuals to mitigate pooling information loss and feed into the decoder.
• Combination layers to fuse motion and content into a unified representation before decoding.
• Deconvolution-based decoder that reconstructs the next frame, aided by residual connections from encoders.
• Multi-frame prediction achieved by iterating the above steps, using previous predictions as inputs for subsequent steps.
• Loss comprised of image-space loss and adversarial loss to produce sharper, more realistic frames.

Experimental results

Research questions

• RQ1 Can separating motion and content in separate encoder pathways improve pixel-level future frame prediction in natural videos?
• RQ2 Does end-to-end training induce natural decomposition of motion and content without supervision?
• RQ3 How does MCnet perform against ConvLSTM baselines and state-of-the-art frame prediction methods on standard video datasets (KTH, Weizmann, UCF-101)?
• RQ4 Do multi-scale residuals improve information preservation and prediction quality across time?

Key findings

• MCnet outperforms a ConvLSTM baseline on long-term predictions and generalizes better to unseen content (KTH and Weizmann datasets).
• The asymmetric motion-content architecture enables natural decomposition of dynamics and layout without explicit supervision.
• On UCF-101, MCnet (single-step) shows clear improvements over the baseline and competitive results relative to the state-of-the-art, with residual variants improving generalization.
• Multi-scale residuals help preserve information across pooling and improve frame sharpness and realism.
• Predictions remain comparatively sharp over longer horizons, capturing periodic motion patterns.
• Qualitative results show MCnet preserving human shapes and motion cues more faithfully than baselines.