[Paper Review] Information-theoretic analysis of generalization capability of learning algorithms

Aolin Xu, Maxim Raginsky|arXiv (Cornell University)|May 22, 2017

Stochastic Gradient Optimization Techniques9 references102 citations

TL;DR

The paper derives upper bounds on learning generalization error via mutual information between input S and output W, and proposes information-theoretic regularization methods (Gibbs and noisy ERM) to control overfitting.

ABSTRACT

We derive upper bounds on the generalization error of a learning algorithm in terms of the mutual information between its input and output. The bounds provide an information-theoretic understanding of generalization in learning problems, and give theoretical guidelines for striking the right balance between data fit and generalization by controlling the input-output mutual information. We propose a number of methods for this purpose, among which are algorithms that regularize the ERM algorithm with relative entropy or with random noise. Our work extends and leads to nontrivial improvements on the recent results of Russo and Zou.

Motivation & Objective

• Provide an information-theoretic understanding of generalization in learning problems.
• Relate generalization error to the input-output mutual information I(S;W).
• Develop algorithmic methods that control I(S;W) to improve generalization.
• Extend prior results to uncountable hypothesis spaces and derive concentration results.
• Discuss adaptive composition and practical algorithm designs using information-theoretic stability.

Proposed method

• Model learning as a randomized channel P_{W|S} from dataset S to hypothesis W.
• Derive upper bounds on gen(mu, P_{W|S}) in terms of I(S;W) under subgaussian loss.
• Present a decoupling lemma linking E[f(X,Y)] to E[f(X̄,Ȳ)] with I(X;Y).
• Show that ERM regularized by I(S;W) leads to Gibbs-like solutions.
• Introduce and analyze Gibbs algorithm as a solution to a regularized risk objective.
• Provide concentration bounds and sample complexity results based on I(S;W).

Experimental results

Research questions

• RQ1 How can generalization error be bounded using information-theoretic quantities like I(S;W)?
• RQ2 Can we extend Russo and Zou’s bounds to uncountable hypothesis spaces and obtain concentration results?
• RQ3 How can regularizing with mutual information (e.g., Gibbs, noisy ERM) control overfitting in practice?
• RQ4 What role does adaptive composition play in preserving generalization under multiple learning stages?
• RQ5 How do these information-theoretic bounds connect to prior knowledge about population risks?

Key findings

• Generalization error can be bounded by sqrt(2σ^2 I(S;W)/n) for σ-subgaussian losses.
• For any loss with subgaussian parameter σ, I(S;W) controls the generalization via Theorem 1.
• Gibbs algorithm and noisy ERM provide practical ways to regulate I(S;W) and thus generalization.
• If the hypothesis space is countable, gen(mu, W) ≤ sqrt(2σ^2 H(W)/n).
• Two-stage procedures and adaptive composition yield overall generalization control by chaining mutual information terms.
• Gibbs algorithm admits prior-informed population-risk guarantees (via choice of Q) and differential privacy implications.