[Paper Review] A Deep Generative Model for Disentangled Representations of Sequential Data.

Yingzhen Li, Stephan Mandt|arXiv (Cornell University)|Mar 8, 2018

Generative Adversarial Networks and Image Synthesis23 references19 citations

TL;DR

This paper proposes a deep variational autoencoder (VAE) that disentangles sequential data into static content and dynamic time-dependent features, enabling controlled generation by conditioning on either component. Experiments show successful content swapping in audio (male-to-female voice conversion) and video (shape and motion manipulation), with stochastic RNNs demonstrating superior efficiency in modeling long sequences.

ABSTRACT

We present a VAE architecture for encoding and generating high dimensional sequential data, such as video or audio. Our deep generative model learns a latent representation of the data which is split into a static and dynamic part, allowing us to approximately disentangle latent time-dependent features (dynamics) from features which are preserved over time (content). This architecture gives us partial control over generating content and dynamics by conditioning on either one of these sets of features. In our experiments on artificially generated cartoon video clips and voice recordings, we show that we can convert the content of a given sequence into another one by such content swapping. For audio, this allows us to convert a male speaker into a female speaker and vice versa, while for video we can separately manipulate shapes and dynamics. Furthermore, we give empirical evidence for the hypothesis that stochastic RNNs as latent state models are more efficient at compressing and generating long sequences than deterministic ones, which may be relevant for applications in video compression.

Motivation & Objective

• To develop a deep generative model that learns disentangled representations of sequential data, such as video and audio.
• To separate latent features into static content and dynamic time-evolving components for independent control.
• To enable conditional generation by manipulating either content or dynamics while preserving the other.
• To evaluate the efficiency of stochastic RNNs versus deterministic RNNs in modeling long sequential data.

Proposed method

• The model uses a VAE framework with a stochastic RNN as the latent state transition model to capture temporal dynamics.
• Latent space is split into two parts: one representing static content and another representing time-evolving dynamics.
• The model is trained end-to-end using variational inference to maximize the evidence lower bound (ELBO).
• Conditional generation is achieved by conditioning on either the content or dynamics latent variables during inference.
• The architecture supports disentangled manipulation by replacing content or dynamics from different sequences.
• Stochastic RNNs are used to model latent transitions, enabling better compression and generation of long sequences.

Experimental results

Research questions

• RQ1 Can a deep generative model effectively disentangle static content from dynamic features in sequential data?
• RQ2 Does conditioning on content or dynamics alone allow for meaningful and controlled generation of new sequences?
• RQ3 How does a stochastic RNN compare to a deterministic RNN in compressing and generating long sequential sequences?
• RQ4 Can the model perform content swapping, such as voice conversion or shape-motion separation, in video and audio?

Key findings

• The model successfully disentangles content and dynamics, enabling independent manipulation of static features and time-evolving components.
• Content swapping in audio allows for effective male-to-female and female-to-male voice conversion while preserving prosody.
• In video generation, the model enables separate manipulation of shapes (content) and motion (dynamics).
• Stochastic RNNs outperform deterministic RNNs in compressing and generating long sequential sequences, supporting the hypothesis of improved modeling efficiency.