[Paper Review] Variational Recurrent Auto-Encoders

Otto Fabius, Joost R. van Amersfoort|arXiv (Cornell University)|Dec 20, 2014

Topic Modeling12 references162 citations

TL;DR

This paper introduces the Variational Recurrent Auto-Encoder (VRAE), a generative model that combines variational inference with recurrent neural networks to learn a continuous, low-dimensional latent representation of sequential data. By using the reparameterization trick and Stochastic Gradient Variational Bayes (SGVB), the VRAE enables efficient, large-scale unsupervised training on time series, allowing for meaningful latent space interpolation and generation of long-duration sequences, while also providing effective initialization for supervised RNNs.

ABSTRACT

In this paper we propose a model that combines the strengths of RNNs and SGVB: the Variational Recurrent Auto-Encoder (VRAE). Such a model can be used for efficient, large scale unsupervised learning on time series data, mapping the time series data to a latent vector representation. The model is generative, such that data can be generated from samples of the latent space. An important contribution of this work is that the model can make use of unlabeled data in order to facilitate supervised training of RNNs by initialising the weights and network state.

Motivation & Objective

• To develop a deep generative model that learns a compact, continuous latent representation of sequential data using variational inference.
• To enable efficient, large-scale unsupervised training of RNNs on time series via the VRAE framework.
• To provide a meaningful initialization of network weights and hidden states for supervised RNNs using pre-trained VRAE parameters.
• To demonstrate the model's ability to generate coherent, long-duration sequences and interpolate between different sequences in the latent space.

Proposed method

• The VRAE uses a bidirectional RNN encoder to map input sequences to a distribution over latent variables, parameterized by mean μ and log-variance log(σ²).
• It applies the reparameterization trick to sample z = μ + σϵ, where ϵ ~ N(0,1), enabling backpropagation through the stochastic sampling process.
• A separate decoder RNN reconstructs the input from the sampled latent vector, with the initial hidden state computed from z using a learned transformation.
• The model is trained end-to-end using Stochastic Gradient Variational Bayes (SGVB), optimizing a lower bound on the log-likelihood that includes a KL divergence term and a reconstruction term.
• For longer sequence modeling, overlapping sequences of 40 time steps are used to capture temporal transitions, improving generation quality.
• t-SNE is used to visualize the 20-dimensional latent representations in 2D, revealing clustering by song type.

Experimental results

Research questions

• RQ1 Can a variational inference framework be effectively combined with RNNs to learn a disentangled, continuous latent representation of sequential data?
• RQ2 How well can the VRAE reconstruct and generate long, coherent sequences from learned latent vectors?
• RQ3 To what extent does the VRAE’s latent space capture meaningful semantic or structural differences between different types of time series (e.g., music pieces)?
• RQ4 Can the VRAE provide a better initialization for supervised RNNs than random or zero initialization?
• RQ5 How does the model’s performance change when using overlapping vs. non-overlapping sequence windows for training?

Key findings

• The VRAE successfully learns a disentangled, low-dimensional latent space where different music pieces occupy distinct, clustered regions, even with only 2D visualization.
• The model achieves a stable training curve with a lower bound on log-likelihood that improves over time, indicating effective optimization.
• Interpolation between latent vectors of different songs produces coherent musical transitions that blend elements of both source songs.
• Long sequences of 1000 time steps (≈50 seconds) can be generated from sampled latent vectors, demonstrating the model’s ability to maintain temporal coherence.
• The t-SNE visualization of 20D latent representations shows that parts of the same song cluster together, indicating the model captures song-specific patterns.
• The VRAE provides a meaningful initialization for standard RNNs, potentially mitigating issues like exploding gradients and improving training efficiency.