[Paper Review] Information bottleneck through variational glasses

Slava Voloshynovskiy, Mouad Kondah+4 more|arXiv (Cornell University)|Dec 2, 2019

Computer Graphics and Visualization Techniques23 references22 citations

TL;DR

This paper introduces a variational decomposition of mutual information within the information bottleneck (IB) framework, unifying supervised, unsupervised, and adversarial generative models under a common theoretical structure. By reinterpreting VAEs and related models through direct decomposition of IB terms, it reveals new connections to methods like $eta$-VAE, InfoVAE, and VAE/GAN, offering improved interpretability and a principled basis for generative compression and anomaly detection.

ABSTRACT

Information bottleneck (IB) principle [1] has become an important element in information-theoretic analysis of deep models. Many state-of-the-art generative models of both Variational Autoencoder (VAE) [2; 3] and Generative Adversarial Networks (GAN) [4] families use various bounds on mutual information terms to introduce certain regularization constraints [5; 6; 7; 8; 9; 10]. Accordingly, the main difference between these models consists in add regularization constraints and targeted objectives. In this work, we will consider the IB framework for three classes of models that include supervised, unsupervised and adversarial generative models. We will apply a variational decomposition leading a common structure and allowing easily establish connections between these models and analyze underlying assumptions. Based on these results, we focus our analysis on unsupervised setup and reconsider the VAE family. In particular, we present a new interpretation of VAE family based on the IB framework using a direct decomposition of mutual information terms and show some interesting connections to existing methods such as VAE [2; 3], beta-VAE [11], AAE [12], InfoVAE [5] and VAE/GAN [13]. Instead of adding regularization constraints to an evidence lower bound (ELBO) [2; 3], which itself is a lower bound, we show that many known methods can be considered as a product of variational decomposition of mutual information terms in the IB framework. The proposed decomposition might also contribute to the interpretability of generative models of both VAE and GAN families and create a new insights to a generative compression [14; 15; 16; 17]. It can also be of interest for the analysis of novelty detection based on one-class classifiers [18] with the IB based discriminators.

Motivation & Objective

• To unify supervised, unsupervised, and adversarial generative models under a common information bottleneck (IB) framework.
• To provide a variational decomposition of mutual information terms that reveals structural similarities across VAE, GAN, and related models.
• To reinterpret the VAE family not via regularization of the ELBO, but through direct decomposition of IB objectives.
• To establish connections between IB, VAEs, $eta$-VAE, AAE, InfoVAE, and VAE/GAN using a unified variational bound.
• To enable improved interpretability and support for applications such as generative compression and one-class novelty detection.

Proposed method

• Proposes a variational decomposition of mutual information $I({f Z};{f C})$ in the IB framework, using a variational distribution $p_{oldsymbol{ heta}}({f c}|{f z})$ to approximate the true conditional $p({f c}|{f z})$.
• Derives a lower bound $I^{ ext{S}}_{oldsymbol{ heta},oldsymbol{ heta}}({f Z};{f C}) = H({f C}) - H_{oldsymbol{ heta},oldsymbol{ heta}}({f C}|{f Z})$ by introducing a variational classifier $p_{oldsymbol{ heta}}({f c}|{f z})$ and leveraging KL divergence as a lower bound.
• Applies the decomposition to unsupervised models by expressing $I({f Z};{f X})$ via $I({f Z};{f X}) = H({f X}) - H_{oldsymbol{ heta},oldsymbol{ heta}}({f X}|{f Z})$, enabling a variational auto-encoding interpretation.
• Uses the variational decomposition to reframe existing models: VAEs, $eta$-VAE, AAE, InfoVAE, and VAE/GAN as special cases of the same IB-based framework.
• Introduces a new objective based on the decomposition that avoids direct regularization of the ELBO, instead focusing on mutual information decomposition.
• Employs the variational lower bound to enable differentiable training and end-to-end optimization across all model types.

Experimental results

Research questions

• RQ1 How can the information bottleneck principle be systematically applied to unify supervised, unsupervised, and adversarial generative models?
• RQ2 What is the role of variational decomposition in revealing structural equivalences between VAE, GAN, and related models?
• RQ3 Can mutual information terms in the IB framework be decomposed in a way that provides a more interpretable and principled alternative to ELBO regularization?
• RQ4 How do known methods like $eta$-VAE and InfoVAE emerge naturally from this unified IB-based variational decomposition?
• RQ5 What are the implications of this framework for generative compression and one-class novelty detection?

Key findings

• The proposed variational decomposition provides a unified theoretical framework that connects supervised, unsupervised, and adversarial models through the information bottleneck principle.
• The method reinterprets VAEs and related models not as ELBO-regularized models, but as instances of mutual information decomposition in the IB framework.
• The lower bound $I^{ ext{S}}_{oldsymbol{ heta},oldsymbol{ heta}}({f Z};{f C})$ is derived via variational approximation of $p({f c}|{f z})$, with the KL divergence term ensuring a valid lower bound.
• The framework naturally recovers known models: $eta$-VAE, AAE, InfoVAE, and VAE/GAN are shown to be special cases of the proposed IB-based variational decomposition.
• The approach enables improved interpretability of generative models and supports applications such as generative compression and one-class novelty detection via IB-based discriminators.
• The analysis demonstrates that mutual information minimization in the IB framework can be achieved through variational decomposition rather than explicit regularization of the ELBO.