[Paper Review] Composing graphical models with neural networks for structured representations and fast inference

Matthew Johnson, David Duvenaud+3 more|arXiv (Cornell University)|Mar 20, 2016

Neural Networks and Applications18 references232 citations

TL;DR

The paper introduces Structured Variational Autoencoders (SVAEs) that combine latent graphical models with neural network observation models, enabling fast inference via recognition networks and efficient message passing.

ABSTRACT

We propose a general modeling and inference framework that composes probabilistic graphical models with deep learning methods and combines their respective strengths. Our model family augments graphical structure in latent variables with neural network observation models. For inference, we extend variational autoencoders to use graphical model approximating distributions with recognition networks that output conjugate potentials. All components of these models are learned simultaneously with a single objective, giving a scalable algorithm that leverages stochastic variational inference, natural gradients, graphical model message passing, and the reparameterization trick. We illustrate this framework with several example models and an application to mouse behavioral phenotyping.

Motivation & Objective

• Combine probabilistic graphical models with deep learning to learn structured latent representations and nonlinear observation models.
• Enable fast, tractable inference by using recognition networks to produce conjugate graphical model potentials.
• Develop SVAE algorithms that integrate stochastic variational inference, message passing, and reparameterization for end-to-end training.
• Demonstrate the framework on tasks such as segmenting and categorizing mouse behavior from depth video.
• Provide scalable, general modeling templates (e.g., warped mixtures, latent LDS, and latent SLDS) that couple discrete/continuous latent structure with neural observation models.

Proposed method

• Propose a general modeling framework that combines latent graphical models with neural network observation likelihoods.
• Use recognition networks to output conjugate exponential-family potentials that can be integrated with graphical model inference.
• Develop the SVAE objective, a lower bound on the mean-field variational objective, optimized via stochastic gradients.
• Leverage conjugate exponential-family structure for efficient gradient calculations, including natural gradients where applicable.
• Provide examples: warped mixtures for flexible clustering, latent linear dynamical systems for video, and latent switching linear dynamical systems for parsing behavior.
• Combine stochastic variational inference, graphical model message passing, and reparameterization tricks in a unified training pipeline.

Experimental results

Research questions

• RQ1 How can we jointly learn structured latent representations and flexible nonlinear observation models?
• RQ2 Can recognition networks yield conjugate graphical model potentials to enable efficient inference with non-conjugate observation models?
• RQ3 How can we perform end-to-end training of models that blend graphical models with deep neural nets?
• RQ4 Do structured latent models (LDS, SLDS) with neural observation models improve tasks like behavior segmentation from video?
• RQ5 What are the computational benefits (e.g., inference speed, optimization stability) of SVAE relative to fully nonconjugate or fully conjugate approaches?

Key findings

• Structured variational autoencoders enable end-to-end training of models that marry graphical-model structure with neural observation models.
• Recognition networks output conjugate potentials allowing fast message-passing inference within CRF-like variational factors.
• SVAE provides a lower bound on the variational objective, with a tractable natural-gradient-informed optimization for latent parameters.
• Experiments on synthetic data and depth video of mice demonstrate effective latent state representations and accurate long-horizon predictions.
• LDS-SVAE and SLDS-SVAE variants show improved learning speed and the emergence of meaningful discrete behavioral states from video data.