[Paper Review] Learning Discrete and Continuous Factors of Data via Alternating Disentanglement

Yeonwoo Jeong, Hyun Oh Song|arXiv (Cornell University)|May 23, 2019

Anomaly Detection Techniques and Applications19 citations

TL;DR

This paper proposes CascadeVAE, a method that alternately disentangles discrete and continuous factors in variational autoencoders by decoupling inference: first inferring discrete factors from continuous latents, then updating the encoder using these discrete factors. It achieves state-of-the-art disentanglement scores and unsupervised classification accuracy on dSprites, MNIST, and FashionMNIST by efficiently penalizing total correlation via information cascading and alternating optimization.

ABSTRACT

We address the problem of unsupervised disentanglement of discrete and continuous explanatory factors of data. We first show a simple procedure for minimizing the total correlation of the continuous latent variables without having to use a discriminator network or perform importance sampling, via cascading the information flow in the $β$-vae framework. Furthermore, we propose a method which avoids offloading the entire burden of jointly modeling the continuous and discrete factors to the variational encoder by employing a separate discrete inference procedure. This leads to an interesting alternating minimization problem which switches between finding the most likely discrete configuration given the continuous factors and updating the variational encoder based on the computed discrete factors. Experiments show that the proposed method clearly disentangles discrete factors and significantly outperforms current disentanglement methods based on the disentanglement score and inference network classification score. The source code is available at https://github.com/snu-mllab/DisentanglementICML19.

Motivation & Objective

• To address the challenge of unsupervised disentanglement of both discrete and continuous factors in complex data like images.
• To overcome the limitation of existing methods that jointly model discrete and continuous factors in a single posterior, which burdens the inference network and degrades disentanglement.
• To develop a method that separately and effectively disentangles discrete and continuous factors through an alternating optimization process.
• To provide an efficient, discriminator-free way to penalize total correlation in the continuous latent space.
• To improve disentanglement performance quantitatively (via disentanglement score) and qualitatively (via latent traversals) on benchmark datasets.

Proposed method

• The method introduces information cascading in the $β$-VAE framework to implicitly penalize total correlation without using a discriminator or importance sampling.
• It formulates an alternating minimization procedure: first, infer the most likely discrete configuration given the current continuous latents; second, update the variational encoder using the inferred discrete factors.
• The approach decouples the modeling burden by treating discrete and continuous factors separately during inference, improving disentanglement quality.
• The total correlation penalty is applied via a cascaded information flow mechanism that avoids explicit computation of high-dimensional mutual information.
• The model uses a standard VAE architecture with a modified training objective that alternates between discrete inference and encoder update steps.
• The method is implemented as CascadeVAE, with code publicly available on GitHub.

Experimental results

Research questions

• RQ1 Can we achieve better disentanglement of discrete and continuous factors by decoupling their inference in a variational autoencoder?
• RQ2 Does alternating optimization between discrete and continuous factor inference improve disentanglement performance compared to joint modeling?
• RQ3 Can total correlation be effectively penalized without using a discriminator or importance sampling in the $β$-VAE framework?
• RQ4 How does the proposed method compare to state-of-the-art disentanglement methods like FactorVAE, JointVAE, and $β$-VAE on quantitative and qualitative benchmarks?
• RQ5 To what extent does the proposed method improve unsupervised classification accuracy of discrete factors?

Key findings

• CascadeVAE achieves a disentanglement score of 81.41 (±9.54) on dSprites with m=10, outperforming JointVAE (68.57 ± 9.19) and other baselines.
• On MNIST, the method achieves a mean unsupervised classification accuracy of 84.19% (±5.02) for discrete factors, significantly exceeding JointVAE's 78.33% (±7.18).
• On FashionMNIST, CascadeVAE achieves a mean classification accuracy of 57.72% (±3.29), surpassing JointVAE's 51.51% (±4.42).
• Latent traversals on MNIST and FashionMNIST show clear disentanglement: continuous dimensions control shape and style, while discrete dimensions control digit identity and object categories.
• On the chairs dataset, the method successfully disentangles categorical shape factors, with discrete latent traversals showing distinct chair types.
• Mutual information between data and latents increases during training, confirming improved representation learning, especially for discrete factors.