[Paper Review] Identifiable Variational Autoencoders via Sparse Decoding.

Gemma E. Moran, Dhanya Sridhar+2 more|arXiv (Cornell University)|Oct 20, 2021

Generative Adversarial Networks and Image Synthesis34 references4 citations

TL;DR

The paper proposes the Sparse VAE, a deep generative model that learns identifiable, sparse latent factors from high-dimensional data by enforcing each observed feature to depend on only a few latent factors. It achieves identifiability under a weak anchor feature assumption and outperforms existing methods in reconstruction accuracy and factor interpretability on simulated and real-world data.

ABSTRACT

We develop the Sparse VAE, a deep generative model for unsupervised representation learning on high-dimensional data. Given a dataset of observations, the Sparse VAE learns a set of latent factors that captures its distribution. The model is sparse in the sense that each feature of the dataset (i.e., each dimension) depends on a small subset of the latent factors. As examples, in ratings data each movie is only described by a few genres; in text data each word is only applicable to a few topics; in genomics, each gene is active in only a few biological processes. We first show that the Sparse VAE is identifiable: given data drawn from the model, there exists a uniquely optimal set of factors. (In contrast, most VAE-based models are not identifiable.) The key assumption behind Sparse-VAE identifiability is the existence of features, where for each factor there exists a feature that depends only on that factor. Importantly, the anchor features do not need to be known in advance. We then show how to fit the Sparse VAE with variational EM. Finally, we empirically study the Sparse VAE with both simulated and real data. We find that it recovers meaningful latent factors and has smaller heldout reconstruction error than related methods.

Motivation & Objective

• To address the lack of identifiability in standard VAEs, which prevents unique recovery of meaningful latent factors.
• To develop a model that learns sparse, disentangled representations where each data feature depends on only a few latent factors.
• To establish conditions under which the latent factors are uniquely identifiable from data without prior knowledge of anchor features.
• To provide a scalable training procedure via variational EM for the proposed model.
• To empirically validate the model’s ability to recover interpretable factors and achieve better reconstruction than baseline methods.

Proposed method

• The Sparse VAE enforces sparsity in the decoding process by restricting each observed feature to depend on only a small subset of latent factors.
• Identifiability is achieved under the assumption that for each latent factor, there exists at least one feature that depends exclusively on it (anchor features), even if these features are unknown a priori.
• The model uses a variational EM framework for training, alternating between posterior inference and model parameter optimization.
• The decoding matrix is constrained to be sparse, with sparsity enforced via a structured prior or regularization during optimization.
• The model learns both the sparse decoding weights and the posterior distribution over latent variables using amortized inference.
• The identifiability proof relies on the structural constraint of sparsity and the existence of anchor features, ensuring a unique optimal factorization.

Experimental results

Research questions

• RQ1 Can a VAE-based model achieve identifiability of latent factors without requiring known anchor features?
• RQ2 How does sparsity in the decoding process affect the interpretability and disentanglement of learned latent factors?
• RQ3 Can the Sparse VAE outperform standard VAEs and other sparse models in reconstruction performance on high-dimensional data?
• RQ4 What conditions on the data structure (e.g., existence of anchor features) are necessary and sufficient for identifiability in the Sparse VAE?
• RQ5 How well does the Sparse VAE recover meaningful, interpretable factors on real-world datasets like text and genomics?

Key findings

• The Sparse VAE is identifiable under the existence of anchor features, even when these features are not known in advance.
• The model achieves smaller held-out reconstruction error compared to related methods on both simulated and real-world datasets.
• The learned latent factors are interpretable, with each feature depending on only a few factors, as seen in applications to ratings, text, and genomics.
• The variational EM training procedure effectively optimizes the model despite the non-convexity of the objective.
• Empirical results show that the model recovers meaningful structures, such as genres in movie ratings or topics in text, consistent with domain knowledge.
• The identifiability result holds under a weak assumption—only the existence of anchor features is required, not their identification.