[Paper Review] Deep Variational Information Bottleneck

Alexander A. Alemi, Ian Fischer+2 more|arXiv (Cornell University)|Dec 1, 2016

Adversarial Robustness in Machine Learning38 references367 citations

TL;DR

This paper introduces a variational approximation to the information bottleneck for neural networks, enabling Deep VIB training that improves generalization and robustness to adversarial attacks.

ABSTRACT

We present a variational approximation to the information bottleneck of Tishby et al. (1999). This variational approach allows us to parameterize the information bottleneck model using a neural network and leverage the reparameterization trick for efficient training. We call this method "Deep Variational Information Bottleneck", or Deep VIB. We show that models trained with the VIB objective outperform those that are trained with other forms of regularization, in terms of generalization performance and robustness to adversarial attack.

Motivation & Objective

• Introduce a variational bound on the information bottleneck objective for deep networks.
• Parameterize the IB objective with neural networks and optimize via reparameterization trick.
• Demonstrate that VIB improves generalization and robustness to adversarial perturbations.

Proposed method

• Formulate the information bottleneck objective as I(Z,Y) - β I(Z,X).
• Model Z as a stochastic encoding p(z|x) and use a variational decoder q(y|z).
• Derive a tractable lower bound L that can be optimized with SGD.
• Apply the reparameterization trick to sample z and backpropagate through stochastic nodes.
• Use a variational prior r(z) to bound I(Z,X) and obtain a KL term in the objective.
• Experiment with neural network encoders/decoders on MNIST and ImageNet to compare with deterministic baselines.

Experimental results

Research questions

• RQ1 Can variational inference provide a scalable optimization of the information bottleneck for deep networks?
• RQ2 Does training with the VIB objective improve generalization compared to regularization baselines?
• RQ3 Is the VIB framework more robust to adversarial perturbations than standard training?
• RQ4 How does compression level (β) affect information in the bottleneck and predictive performance?

Key findings

• VIB achieves lower test error (1.13%) on permutation-invariant MNIST compared to several regularization baselines.
• With β in an intermediate range, Monte Carlo averaging over z improves robustness without sacrificing accuracy.
• VIB models show increased robustness to adversarial attacks on MNIST under FGS and L2 optimization attacks.
• Increasing β leads to smaller I(Z,X) while maintaining predictive power on training data, illustrating the IB tradeoff.
• On ImageNet, VIB demonstrates adversarial resistance not present in deterministic baselines.