[Paper Review] Machine Learning using the Variational Predictive Information Bottleneck with a Validation Set

Sayandev Mukherjee|arXiv (Cornell University)|Nov 6, 2019

Statistical Mechanics and Entropy3 references4 citations

TL;DR

This paper extends the predictive information bottleneck framework to machine learning with a validation set by formulating model training as an information processing optimization that balances predictive power on future data, compression of training data, and validation set performance. The key contribution is a variational inference objective that integrates training, validation, and future data through a joint information bottleneck criterion, yielding a Bayesian-like update rule when optimization succeeds.

ABSTRACT

Zellner (1988) modeled statistical inference in terms of information processing and postulated the Information Conservation Principle (ICP) between the input and output of the information processing block, showing that this yielded Bayesian inference as the optimum information processing rule. Recently, Alemi (2019) reviewed Zellner's work in the context of machine learning and showed that the ICP could be seen as a special case of a more general optimum information processing criterion, namely the Predictive Information Bottleneck Objective. However, Alemi modeled the model training step in machine learning as using training and test data sets only, and did not account for the use of a validation data set during training. The present note is an attempt to extend Alemi's information processing formulation of machine learning, and the predictive information bottleneck objective for model training, to the widely-used scenario where training utilizes not only a training but also a validation data set.

Motivation & Objective

• To extend the predictive information bottleneck framework to include validation sets in machine learning, which are commonly used in practice but absent in prior information-theoretic formulations.
• To formalize model training as an information processing task that maximizes mutual information between model parameters and future data, while constraining information from training and validation data.
• To derive a variational approximation that enables tractable optimization of the joint information bottleneck objective involving past, validation, and future data.
• To show that successful optimization under the new objective yields a Bayesian update rule that incorporates training, validation, and future data.

Proposed method

• Proposes a joint information bottleneck objective that maximizes mutual information between model parameters and future data, while constraining information from both training and validation data via Lagrange multipliers.
• Derives an equivalent unconstrained optimization problem using the Markov property and information-theoretic identities, leading to a minimization of a weighted sum of conditional mutual informations.
• Applies two variational approximations: one for the posterior over model parameters and another for the likelihoods of training and validation data, enabling tractable optimization.
• Introduces a variational distribution for model parameters that combines prior, training likelihood, and validation likelihood with temperature-like hyperparameters β and γ.
• Derives the optimal variational posterior as a normalized product of the prior, training likelihood raised to β, and validation likelihood raised to γ, with a partition function Zβ,γ.
• Demonstrates that when information constraints are satisfied, the optimal update rule reduces to Bayesian inference with a joint likelihood incorporating training and validation data.

Experimental results

Research questions

• RQ1 How can the predictive information bottleneck framework be extended to include a validation set in machine learning training?
• RQ2 What is the information-theoretic objective that balances predictive performance on future data with compression of training and validation data?
• RQ3 How does the inclusion of a validation set affect the variational approximation and optimization of the model posterior?
• RQ4 Under what conditions does the proposed objective reduce to Bayesian inference?
• RQ5 What is the role of the hyperparameters β and γ in controlling the trade-off between training, validation, and future data information?

Key findings

• The proposed objective function is upper-bounded by a variational objective that depends on the true data distributions and a variational posterior over model parameters.
• The optimal variational posterior is derived as a normalized product of the prior, training likelihood to the power β, and validation likelihood to the power γ, with a partition function Zβ,γ.
• When the information constraints on training and validation data are satisfied (I(θ; xP) > I′0 and I(θ; xV|xP) > I′′0), the optimal update rule becomes Bayesian inference with a joint likelihood over training and validation data.
• The method provides a principled information-theoretic foundation for using validation sets in model training, showing that such practice naturally emerges from an information processing perspective.
• The framework generalizes Alemi’s predictive information bottleneck to include validation data, offering a unified view of training, validation, and generalization in terms of mutual information.
• The approach maintains consistency with Bayesian inference under successful optimization, suggesting that validation-based hyperparameter tuning aligns with optimal information processing.