[Paper Review] Sequential Neural Models with Stochastic Layers

M. Fraccaro, Søren Kaae Sønderby+2 more|arXiv (Cornell University)|May 24, 2016

Music and Audio Processing158 citations

TL;DR

The paper introduces stochastic recurrent neural networks (SRNNs) that combine a deterministic RNN with a nonlinear state space model, enabling structured variational inference and achieving state-of-the-art results on Blizzard and TIMIT speech datasets while performing competitively on polyphonic music.

ABSTRACT

How can we efficiently propagate uncertainty in a latent state representation with recurrent neural networks? This paper introduces stochastic recurrent neural networks which glue a deterministic recurrent neural network and a state space model together to form a stochastic and sequential neural generative model. The clear separation of deterministic and stochastic layers allows a structured variational inference network to track the factorization of the model's posterior distribution. By retaining both the nonlinear recursive structure of a recurrent neural network and averaging over the uncertainty in a latent path, like a state space model, we improve the state of the art results on the Blizzard and TIMIT speech modeling data sets by a large margin, while achieving comparable performances to competing methods on polyphonic music modeling.

Motivation & Objective

• Motivate efficient propagation of uncertainty in latent sequential representations.
• Propose a stochastic recurrent architecture that cleanly separates deterministic and stochastic layers.
• Develop a structured variational inference network that mirrors the posterior factorization of the model.
• Demonstrate state-of-the-art performance on speech modeling and competitive results on music modeling.

Proposed method

• Stack a nonlinear state-space model on top of a gated RNN (GRU-based) to form a stochastic recurrent neural network (SRNN).
• Factor the joint generative model p_theta(x1:T, z1:T, d1:T) into p(x_t|z_t,d_t) p(z_t|z_{t-1},d_t) p(d_t|d_{t-1},u_t).
• Use a structured inference network q_phi(z1:T, d1:T|x1:T, u1:T) that leverages backward information via a backward RNN to approximate z_t given future observations and states.
• Employ variational inference with an ELBO (F(theta, phi)) and the reparameterization trick for gradient-based optimization.
• Introduce a residual-based parameterization (Res_q) to improve the inference network by incorporating predictive prior dynamics through samples from q_phi*(z_{t-1}).
• Provide both smoothing (backward) and filtering variants for the inference network.

Experimental results

Research questions

• RQ1 Can a stochastic latent layer atop an RNN effectively model uncertainty in sequential data while preserving the benefits of gated nonlinearities?
• RQ2 Does incorporating a nonlinear state-space layer and a structured variational posterior improve predictive performance on speech and music data?
• RQ3 How should the inference network be designed to exploit future information and maintain tractable, efficient training?
• RQ4 What are the practical benefits of separating deterministic and stochastic components for posterior inference in sequential models?

Key findings

• SRNN achieves state-of-the-art or competitive performance on Blizzard and TIMIT speech modeling tasks.
• A structured variational posterior that integrates future information (smoothing) improves the ELBO and predictive likelihood compared to filtering alone.
• Using a residual parameterization in the inference network further improves performance in most cases.
• SRNN attains competitive results on polyphonic music datasets, comparable to several contemporary methods.
• Separating the deterministic RNN core from the stochastic state-space layer simplifies inference and training while preserving modeling power.