[Paper Review] Unsupervised Learning of Disentangled Representations from Video

Emily Denton, Vighnesh Birodkar|arXiv (Cornell University)|May 31, 2017

Generative Adversarial Networks and Image Synthesis228 citations

TL;DR

DrNet learns content (time-invariant) and pose (time-varying) disentangled representations from video using a novel adversarial loss, enabling long-range frame prediction and effective classification from either component.

ABSTRACT

We present a new model DrNET that learns disentangled image representations from video. Our approach leverages the temporal coherence of video and a novel adversarial loss to learn a representation that factorizes each frame into a stationary part and a temporally varying component. The disentangled representation can be used for a range of tasks. For example, applying a standard LSTM to the time-vary components enables prediction of future frames. We evaluate our approach on a range of synthetic and real videos, demonstrating the ability to coherently generate hundreds of steps into the future.

Motivation & Objective

• Motivate unsupervised learning of robust video representations without labels.
• Factor video frames into a stationary content component and a dynamic pose component.
• Introduce an adversarial loss to ensure pose carries no clip-specific content information.
• Demonstrate long-range frame prediction and classification using the disentangled features.

Proposed method

• Two encoders produce content (E_c) and pose (E_p) representations per frame.
• A decoder (D) predicts future frames from concatenated content and future pose features.
• An adversarial discriminator (C) enforces that pose features cannot reveal clip identity.
• A similarity loss encourages content features to vary slowly over time.
• An overall objective combines reconstruction, similarity, and adversarial terms with tunable weights.

Experimental results

Research questions

• RQ1 Can video frames be factorized into a time-invariant content and a time-varying pose without supervision?
• RQ2 Does adversarial training on pose features enforce content/pose disentanglement while preserving predictive reconstruction?
• RQ3 Can the disentangled representations support accurate long-range video prediction and downstream classification tasks?

Key findings

• The model learns a clean content/pose factorization demonstrated on synthetic and real videos.
• Long-range frame predictions up to hundreds of steps are feasible using a simple LSTM over pose features with fixed content from the last observed frame.
• Content features support semantic classification, while pose features support action prediction.
• Adversarial loss is crucial for enforcing disentanglement; removing it degrades content pose separation and classification performance.
• On NORB, content features achieve high accuracy when β=0.1, and pose features perform differently under β settings (see table).
• The approach yields competitive or favorable qualitative results compared with baselines on real video (KTH) and synthetic data.