[Paper Review] Reasoning About Generalization via Conditional Mutual Information

Thomas Steinke, Lydia Zakynthinou|arXiv (Cornell University)|Jan 24, 2020

Stochastic Gradient Optimization Techniques74 references17 citations

TL;DR

This paper introduces a unifying information-theoretic framework for analyzing machine learning generalization using Conditional Mutual Information (CMI), showing that bounded CMI across diverse settings—such as VC dimension, differential privacy, and compression schemes—implies generalization. The key contribution is a formal link between CMI and generalization error, enabling tighter, more adaptable bounds across multiple learning paradigms.

ABSTRACT

We provide an information-theoretic framework for studying the generalization properties of machine learning algorithms. Our framework ties together existing approaches, including uniform convergence bounds and recent methods for adaptive data analysis. Specifically, we use Conditional Mutual Information (CMI) to quantify how well the input (i.e., the training data) can be recognized given the output (i.e., the trained model) of the learning algorithm. We show that bounds on CMI can be obtained from VC dimension, compression schemes, differential privacy, and other methods. We then show that bounded CMI implies various forms of generalization.

Motivation & Objective

• To unify disparate approaches to generalization in machine learning—such as uniform convergence, differential privacy, and compression—under a single information-theoretic framework.
• To formalize how the conditional mutual information (CMI) between training data and model output quantifies generalization performance.
• To demonstrate that bounded CMI implies generalization, regardless of the underlying algorithmic mechanism.
• To provide a systematic method for deriving CMI bounds from known generalization techniques like VC dimension and stability notions.
• To extend the framework to adaptive data analysis and non-linear loss functions, enhancing its practical applicability.

Proposed method

• Define the conditional mutual information (CMI) between the training data Z and the model A(Z), given the data distribution D, to measure how much the model reveals about the training data.
• Use CMI as a proxy for generalization error, showing that bounded CMI implies small expected difference between empirical and true risk.
• Derive CMI bounds from known generalization mechanisms: VC dimension, differential privacy, compression schemes, and uniform stability.
• Introduce evaluated CMI (eCMI) as a practical proxy for CMI in finite-sample settings, enabling empirical estimation.
• Apply concentration inequalities and KL divergence to bound CMI and relate it to loss functions like squared error and hinge loss.
• Establish a connection between uniform stability and eCMI, showing that stable algorithms yield low eCMI under continuous data distributions.

Experimental results

Research questions

• RQ1 Can conditional mutual information (CMI) serve as a unifying measure for generalization across diverse learning algorithms?
• RQ2 How can CMI be bounded using existing generalization techniques such as VC dimension, differential privacy, and compression?
• RQ3 Does bounded CMI imply generalization in both linear and non-linear loss settings?
• RQ4 Can the framework be extended to adaptive data analysis and non-i.i.d. data settings?
• RQ5 How does evaluated CMI (eCMI) relate to uniform stability, and can it yield tighter generalization bounds?

Key findings

• Bounded CMI between training data and model output implies generalization, with the generalization error upper-bounded by a function of the CMI value.
• CMI bounds can be derived from VC dimension, showing that low-complexity hypothesis classes yield low CMI and thus good generalization.
• Differential privacy implies bounded CMI, linking privacy and generalization through information-theoretic measures.
• Compression schemes also yield bounded CMI, demonstrating that model simplicity translates to information-theoretic robustness.
• For uniformly stable algorithms, the evaluated CMI (eCMI) is bounded by O(γ²n²/σ²) for Gaussian noise, with a resulting generalization error bound of O(γ + 1/n).
• The framework extends to non-linear losses, including squared error and hinge loss, with explicit bounds derived for these cases.