[Paper Review] Auxiliary Deep Generative Models

Lars Maaløe, Casper Kaae Sønderby+2 more|arXiv (Cornell University)|Feb 17, 2016

Generative Adversarial Networks and Image Synthesis21 references154 citations

TL;DR

The paper introduces Auxiliary Deep Generative Models (ADGM) and Skip Deep Generative Models (SDGM) that use auxiliary latent variables to make variational posteriors more expressive, enabling end-to-end training and state-of-the-art semi-supervised results on MNIST, SVHN, and NORB.

ABSTRACT

Deep generative models parameterized by neural networks have recently achieved state-of-the-art performance in unsupervised and semi-supervised learning. We extend deep generative models with auxiliary variables which improves the variational approximation. The auxiliary variables leave the generative model unchanged but make the variational distribution more expressive. Inspired by the structure of the auxiliary variable we also propose a model with two stochastic layers and skip connections. Our findings suggest that more expressive and properly specified deep generative models converge faster with better results. We show state-of-the-art performance within semi-supervised learning on MNIST, SVHN and NORB datasets.

Motivation & Objective

• Extend deep generative models with auxiliary variables to improve variational approximation.
• Demonstrate that auxiliary variables yield faster convergence and better performance in semi-supervised learning.
• Introduce a two-layer stochastic model with skip connections (SDGM) and evaluate on standard benchmarks.

Proposed method

• Introduce auxiliary latent variables a to p(x,z,a) such that p(x,z,a)=p(a|x,z)p(x,z) to keep the model invariant to marginalization over a.
• Derive a variational lower bound that incorporates a, enabling q(a|x) and q(z|a,x) to form a richer posterior (AVAE/ADGM framework).
• Extend to semi-supervised learning by adding a class latent y with q(y|a,x) and pθ(a|z,y,x), pθ(x|z,y).
• Define the inference model qφ(a|x) qφ(y|a,x) qφ(z|a,y,x) and use reparameterization for backpropagation.
• Propose the Skip Deep Generative Model (SDGM) by reversing the arrow to create a two-layer generative model with skip connections.
• Train end-to-end with Adam, using a combined objective balancing generative and discriminative terms (α weight).
• Evaluate on toy distributions, MNIST, SVHN, and NORB, comparing AVAE/ADGM/SDGM to baseline VAEs and semi-supervised methods.

Experimental results

Research questions

• RQ1 Can auxiliary variables in variational distributions better approximate complex posteriors in deep generative models?
• RQ2 Do ADGM and SDGM improve convergence speed and semi-supervised classification accuracy on standard benchmarks?
• RQ3 How does the auxiliary-variable approach perform compared to other semi-supervised methods (VAT, Ladder, M2) on MNIST, SVHN, and NORB?

Key findings

• ADGM and SDGM achieve state-of-the-art semi-supervised performance on MNIST, SVHN, and NORB among reported methods.
• On MNIST with 100 labeled examples, ADGM reaches 0.96% error (±0.02) and SDGM 1.32% (±0.07) with competitive or better results than several baselines.
• On SVHN, ADGM yields 22.86% error and SDGM yields 16.61% error, outperforming VAT and comparable to Ladder in some settings.
• On NORB, ADGM achieves 10.06% (±0.05) and SDGM 9.40% (±0.04), with SDGM often more stable than ADGM for Gaussian inputs.
• Toy experiments show auxiliary variables enable modeling multi-modal/posterior distributions beyond Gaussian, improving the variational bound.
• SDGM converges faster than ADGM on synthetic semi-supervised tasks and demonstrates clearer separation of data manifolds.