[Paper Review] How (Not) To Train Your Neural Network Using the Information Bottleneck Principle.

Rana Ali Amjad, Bernhard C. Geiger|arXiv (Cornell University)|Feb 27, 2018

Stochastic Gradient Optimization Techniques21 citations

TL;DR

This paper demonstrates that training deep neural networks via the information bottleneck (IB) principle is fundamentally flawed for deterministic networks, as the IB functional becomes infinite and invariant under bijections, rendering optimization ill-posed and incapable of capturing key classification properties. The authors argue that only stochastic networks or modified cost functions resolve these issues, implying that reported IB successes stem from such adaptations, not the IB framework itself.

ABSTRACT

In this theory paper, we investigate training deep neural networks (DNNs) for classification via minimizing the information bottleneck (IB) functional. We show that, even if the joint distribution between continuous feature variables and the discrete class variable is known, the resulting optimization problem suffers from two severe issues: First, for deterministic DNNs, the IB functional is infinite for almost all weight matrices, making the optimization problem ill-posed. Second, the invariance of the IB functional under bijections prevents it from capturing desirable properties for classification, such as robustness, architectural simplicity, and simplicity of the learned representation. We argue that these issues are partly resolved for stochastic DNNs, DNNs that include a (hard or soft) decision rule, or by replacing the IB functional with related, but more well-behaved cost functions. We conclude that recent successes reported about training DNNs using the IB framework must be attributed to such solutions. As a side effect, our results imply limitations of the IB framework for the analysis of DNNs.

Motivation & Objective

• To investigate the feasibility of training deep neural networks via minimization of the information bottleneck (IB) functional.
• To identify fundamental theoretical issues in applying the IB principle to deterministic deep neural networks.
• To analyze why the IB functional fails to capture desirable properties like robustness and representation simplicity in classification tasks.
• To explore whether stochasticity or alternative cost functions can resolve the IB framework's limitations.
• To clarify the role of the IB framework in analyzing deep neural network training, given its theoretical shortcomings.

Proposed method

• Analyzing the IB functional under known joint distributions between continuous features and discrete classes.
• Proving that the IB functional is infinite for almost all weight matrices in deterministic DNNs, rendering optimization ill-posed.
• Demonstrating the invariance of the IB functional under bijective transformations of network weights, which undermines its ability to enforce meaningful inductive biases.
• Evaluating the impact of stochasticity in DNNs, such as hard or soft decision rules, on the well-posedness of the IB optimization problem.
• Proposing alternative cost functions that are better-behaved than the standard IB functional for training DNNs.
• Using theoretical analysis to compare the IB functional with its modified variants in terms of optimization stability and inductive bias.

Experimental results

Research questions

• RQ1 Why does the information bottleneck functional become infinite for deterministic deep neural networks?
• RQ2 How does the invariance of the IB functional under bijections affect its ability to learn robust and simple representations?
• RQ3 What conditions must be met for the IB framework to yield well-posed optimization in deep learning?
• RQ4 Can stochasticity in neural networks resolve the theoretical flaws of the IB approach?
• RQ5 To what extent do recent successes in IB-based training rely on modifications beyond the standard IB functional?

Key findings

• The information bottleneck functional is infinite for almost all weight matrices in deterministic deep neural networks, making the optimization problem ill-posed.
• The IB functional is invariant under bijective transformations of the network weights, preventing it from capturing architectural simplicity or robustness.
• Stochastic deep neural networks, particularly those with hard or soft decision rules, resolve the ill-posedness and invariance issues of the IB functional.
• The reported success of IB-based training methods is likely due to the use of stochasticity or alternative cost functions, not the IB principle itself.
• The standard IB framework is fundamentally limited for analyzing or training deep neural networks, especially in deterministic settings.
• Alternative cost functions that are more well-behaved than the IB functional are necessary for stable and meaningful training.