[Paper Review] Understanding disentangling in $β$-VAE

Christopher Burgess, Irina Higgins+5 more|arXiv (Cornell University)|Apr 10, 2018

Generative Adversarial Networks and Image SynthesisComputer Science277 citations

TL;DR

This paper analyzes why β-VAE learns disentangled representations and proposes a capacity-increase training regime that improves disentanglement and reconstruction quality. It links β-VAE to information bottleneck and demonstrates a controlled capacity growth strategy.

ABSTRACT

We present new intuitions and theoretical assessments of the emergence of disentangled representation in variational autoencoders. Taking a rate-distortion theory perspective, we show the circumstances under which representations aligned with the underlying generative factors of variation of data emerge when optimising the modified ELBO bound in $β$-VAE, as training progresses. From these insights, we propose a modification to the training regime of $β$-VAE, that progressively increases the information capacity of the latent code during training. This modification facilitates the robust learning of disentangled representations in $β$-VAE, without the previous trade-off in reconstruction accuracy.

Motivation & Objective

• Motivate unsupervised disentangled representation learning and its potential benefits for transfer and generalization.
• Theoretically relate β-VAE to information bottleneck and explain axis-aligned disentanglement.
• Propose and validate a training modification that gradually increases latent capacity to improve disentanglement and reconstructions.

Proposed method

• Explain β-VAE objective and its relation to the information bottleneck concept.
• Interpret the KL term as a bound on information transmission through latent channels.
• Use a simplified factor-conditional generator to study capacity vs. disentanglement.
• Introduce a capacity-control objective that gradually increases target KL from zero to a final value.
• Empirically evaluate on dSprites, coloured dSprites, and 3D Chairs with latent traversals and reconstructions.

Experimental results

Research questions

• RQ1 Why does β-VAE tend to learn axis-aligned, disentangled representations?
• RQ2 How does the information bottleneck pressure shape latent axes and factor disentanglement?
• RQ3 Can progressively increasing latent capacity during training improve disentanglement without sacrificing reconstruction quality?
• RQ4 How does the proposed capacity-controlled training affect qualitatively disentangled factors on standard datasets?
• RQ5 Do the learned latent axes correspond to human-interpretable factors of variation across datasets?

Key findings

• β-VAE induces locality-preserving latent representations and axis-alignment with underlying factors of variation.
• Posterior capacity constraints bias encoding toward factors that most improve data log-likelihood, leading to disentanglement.
• A controlled capacity increase yields robust disentangling and better reconstructions than fixed β objective.
• On coloured dSprites and 3D Chairs, latent traversals show factors such as position, scale, shape, rotation, and color are independently encoded.
• The capacity-increase approach enables progressively richer representations while maintaining disentanglement across factors.