[論文レビュー] Learning to Decompose and Disentangle Representations for Video Prediction

Jun-Ting Hsieh, Bingbin Liu+3 more|arXiv (Cornell University)|Jun 11, 2018

Generative Adversarial Networks and Image Synthesis参考文献 45被引用数 107

ひとこと要約

DDPAEは、ビデオを自動的にコンポーネントに分解し、それぞれを低次元の時系列ダイナミクスに分解して、ピクセルから将来のフレームを予測するよう設計された、明示的な監督なしで動作するフレームワークです。

ABSTRACT

Our goal is to predict future video frames given a sequence of input frames. Despite large amounts of video data, this remains a challenging task because of the high-dimensionality of video frames. We address this challenge by proposing the Decompositional Disentangled Predictive Auto-Encoder (DDPAE), a framework that combines structured probabilistic models and deep networks to automatically (i) decompose the high-dimensional video that we aim to predict into components, and (ii) disentangle each component to have low-dimensional temporal dynamics that are easier to predict. Crucially, with an appropriately specified generative model of video frames, our DDPAE is able to learn both the latent decomposition and disentanglement without explicit supervision. For the Moving MNIST dataset, we show that DDPAE is able to recover the underlying components (individual digits) and disentanglement (appearance and location) as we would intuitively do. We further demonstrate that DDPAE can be applied to the Bouncing Balls dataset involving complex interactions between multiple objects to predict the video frame directly from the pixels and recover physical states without explicit supervision.

研究の動機と目的

• Motivate reducing prediction complexity by decomposing high-dimensional video into components.
• Automatically discover decomposed components and their low-dimensional temporal dynamics without supervision.
• Show that decomposition and disentanglement improve future-frame prediction on Moving MNIST and Bouncing Balls.

提案手法

• Formulate DDPAE as a structured probabilistic model with deep parameterization.
• Decompose video into N components with shared content and low-dimensional pose per component.
• Predict low-dimensional pose dynamics for each component and reconstruct frames via a frame decoder with spatial transformers.
• Infer latent variables with a variational autoencoder framework and optimize ELBO.

実験結果

リサーチクエスチョン

• RQ1 Can automatic decomposition of video into components with disentangled, low-dimensional dynamics facilitate more accurate future frame prediction?
• RQ2 Does learning both decomposition and disentanglement improve predictions on datasets with moving digits and interacting objects?
• RQ3 Can the model handle interdependent components and unknown numbers of objects?
• RQ4 How well does DDPAE recover interpretable components (e.g., digits, balls) from pixels without supervision.

主な発見

Model	BCE	MSE
Shi et al. [45]	367.2	-
Srivastava et al. [33]	341.2	-
Brabandere et al. [5]	285.2	-
Patraucean et al. [26]	262.6	-
Ghosh et al. [10]	241.8	167.9
Kalchbrenner et al. [15]	87.6	-
MCNet [39]	1308.2	173.2
DRNet [6]	862.7	163.9
Ours w/o Decomposition	325.5	77.6
Ours w/o Disentanglement	296.1	65.6
Ours (DDPAE)	223.0	38.9

• DDPAE significantly outperforms baselines without decomposition or without disentanglement on Moving MNIST (lower BCE and MSE).
• The model learns to separate digits into components and disentangle appearance (content) from position (pose) automatically.
• On Bouncing Balls, DDPAE predicts complex interactions (collisions) directly from pixels and recovers physical properties without explicit state modeling.
• DDPAE demonstrates robustness to unknown/variable numbers of components by allocating extra components as empty when unnecessary.
• Interdependent-component modeling improves velocity prediction during collisions compared to independent components.