[Paper Review] General Information Bottleneck Objectives and their Applications to Machine Learning

Sayandev Mukherjee|arXiv (Cornell University)|Dec 12, 2019

Neural Networks and Applications12 references4 citations

TL;DR

This paper introduces a unified family of General Information Bottleneck Objectives (IBOs) that generalize the Information Bottleneck Principle (IBP) and Predictive Information Bottleneck Principle (PIBP), resolving a contradiction between IBP/PIBP (which maximize mutual information between model parameters and test data) and recent results (which minimize this mutual information). It derives a new IBO accounting for the fact that model parameters are selected to minimize empirical loss, and provides a variational upper bound for optimization, reconciling conflicting theoretical recommendations.

ABSTRACT

We view the Information Bottleneck Principle (IBP: Tishby et al., 1999; Schwartz-Ziv and Tishby, 2017) and Predictive Information Bottleneck Principle (PIBP: Still et al., 2007; Alemi, 2019) as special cases of a family of general information bottleneck objectives (IBOs). Each IBO corresponds to a particular constrained optimization problem where the constraints apply to: (a) the mutual information between the training data and the learned model parameters or extracted representation of the data, and (b) the mutual information between the learned model parameters or extracted representation of the data and the test data (if any). The heuristics behind the IBP and PIBP are shown to yield different constraints in the corresponding constrained optimization problem formulations. We show how other heuristics lead to a new IBO, different from both the IBP and PIBP, and use the techniques from (Alemi, 2019) to derive and optimize a variational upper bound on the new IBO. We then apply the theory of general IBOs to resolve the seeming contradiction between, on the one hand, the recommendations of IBP and PIBP to maximize the mutual information between the model parameters and test data, and on the other, recent information-theoretic results (see Xu and Raginsky, 2017) suggesting that this mutual information should be minimized. The key insight is that the heuristics (and thus the constraints in the constrained optimization problems) of IBP and PIBP are not applicable to the scenario analyzed by (Xu and Raginsky, 2017) because the latter makes the additional assumption that the parameters of the trained model have been selected to minimize the empirical loss function. Aided by this insight, we formulate a new IBO that accounts for this property of the parameters of the trained model, and derive and optimize a variational bound on this IBO.

Motivation & Objective

• To resolve the apparent contradiction between the Information Bottleneck Principle (IBP) and Predictive IBP (PIBP), which advocate maximizing mutual information between model parameters and test data, and recent information-theoretic results (Xu and Raginsky, 2017) that suggest minimizing this mutual information.
• To formalize IBP and PIBP as special cases within a broader family of constrained optimization problems defined by mutual information constraints on model parameters and training/test data.
• To identify the heuristic assumptions underlying IBP and PIBP that make them inapplicable to the scenario analyzed by Xu and Raginsky (2017), where model parameters are assumed to minimize empirical loss.
• To propose a new Information Bottleneck Objective (IBO) that explicitly incorporates the constraint that model parameters are chosen to minimize empirical loss on the training set.
• To derive and optimize a variational upper bound for the new IBO using techniques from Alemi (2019), enabling practical application in machine learning.

Proposed method

• Formalizes IBP and PIBP as members of a general family of Information Bottleneck Objectives (IBOs) of the form $ I_1 - \nu I_2 $, where $ I_1 $ and $ I_2 $ are mutual informations between model parameters and training/test data, respectively.
• Identifies that IBP and PIBP correspond to different constrained optimization problems with distinct constraints on $ I(t;\bm{x_P}) $ and $ I(t;\bm{x_F}) $, based on their underlying heuristics.
• Demonstrates that the IBP and PIBP heuristics do not apply to the scenario in Xu and Raginsky (2017), which assumes model parameters are selected to minimize empirical loss.
• Proposes a new IBO: $ \max_{p(\theta|\bm{x_P})} \left[ I(\theta;\bm{x_P}|\bm{x_F}) - \beta I(\theta;\bm{x_P}) \right] $, subject to $ L(\theta,\bm{x_P}) \leq \epsilon $, where $ \beta \geq 1 $.
• Derives a variational upper bound for the new IBO using the same framework as Alemi (2019), enabling optimization via variational inference.
• Applies the bound $ I(\theta;\bm{x_P}|\bm{x_F}) - \beta I(\theta;\bm{x_P}) \leq -\beta H(\bm{x_P}) - \mathbb{E}_{p(\phi)p(\bm{x_P}|\phi)} \log Z_\beta(\bm{x_P}) $, with $ Z_\beta $ defined via auxiliary distributions $ q(\theta) $ and $ q(\bm{x_P}|\theta) $.

Experimental results

Research questions

• RQ1 Why do IBP and PIBP recommend maximizing mutual information between model parameters and test data, while recent generalization bounds suggest minimizing it?
• RQ2 What assumptions underlie the IBP and PIBP heuristics that make them inapplicable to the scenario in Xu and Raginsky (2017)?
• RQ3 How can a new Information Bottleneck Objective be formulated that accounts for the fact that model parameters are selected to minimize empirical loss?
• RQ4 Can a variational upper bound be derived and optimized for this new IBO to enable practical use in machine learning?
• RQ5 How does the new IBO reconcile the conflicting recommendations of maximizing versus minimizing mutual information in generalization theory?

Key findings

• The IBP and PIBP are shown to be special cases of a broader family of constrained optimization problems defined by mutual information constraints on model parameters and training/test data.
• The key insight is that the IBP and PIBP heuristics do not apply to the scenario in Xu and Raginsky (2017), which assumes model parameters are chosen to minimize empirical loss, a condition not assumed in IBP or PIBP.
• A new IBO is proposed: $ \max_{p(\theta|\bm{x_P})} \left[ I(\theta;\bm{x_P}|\bm{x_F}) - \beta I(\theta;\bm{x_P}) \right] $, with $ \beta \geq 1 $, that incorporates the empirical loss minimization constraint.
• The paper derives a tight variational upper bound for the new IBO: $ I(\theta;\bm{x_P}|\bm{x_F}) - \beta I(\theta;\bm{x_P}) \leq -\beta H(\bm{x_P}) - \mathbb{E}_{p(\phi)p(\bm{x_P}|\phi)} \log Z_\beta(\bm{x_P}) $, which enables optimization via variational inference.
• The new IBO resolves the contradiction between IBP/PIBP and recent generalization bounds by explicitly modeling the fact that model parameters are selected to minimize empirical loss on the training set.
• The proposed framework provides a unified treatment of existing IBP and PIBP objectives and offers a principled way to derive new objectives that reflect realistic training dynamics.