[Paper Review] A Note on the PAC Bayesian Theorem

Andreas Maurer|ArXiv.org|Nov 30, 2004

Machine Learning and Algorithms8 references97 citations

TL;DR

This paper tightens the PAC Bayesian theorem by deriving improved exponential moment inequalities for [0,1]-valued i.i.d. random variables, reducing the logarithmic dependence on sample size from ln(2n) to ln(2√n) in the generalization bound. The key contribution is a tighter PAC-Bayesian bound with a halved logarithmic term, significantly improving confidence bounds for learning algorithms without increasing computational cost.

ABSTRACT

We prove general exponential moment inequalities for averages of [0,1]-valued iid random variables and use them to tighten the PAC Bayesian Theorem. The logarithmic dependence on the sample count in the enumerator of the PAC Bayesian bound is halved.

Motivation & Objective

• To improve the PAC-Bayesian generalization bound by reducing the logarithmic term in the confidence bound.
• To derive tighter exponential moment inequalities for the Kullback-Leibler divergence between empirical and true means of [0,1]-valued i.i.d. random variables.
• To show that the square root dependence on n in the upper bound is optimal for Bernoulli variables.
• To apply the improved bounds to the PAC-Bayesian theorem, yielding a tighter generalization error bound with reduced failure probability.

Proposed method

• Derives an upper bound on the moment generating function of n times the KL divergence between the empirical mean and true mean: E[exp(n KL(M(X), μ))] ≤ 2√n for n ≥ 8.
• Establishes a matching lower bound: E[exp(n KL(M(X), μ))] ≥ √n for n ≥ 2, showing the √n dependence is optimal.
• Uses Jensen’s inequality and convexity of the KL divergence and exponential function to bound the posterior-dependent expectation in the PAC-Bayesian framework.
• Applies Markov’s inequality to the moment generating function to derive a high-probability bound on the KL divergence between posterior and true risk.
• Introduces a data-dependent posterior Q_S via exponential weighting based on empirical loss, and uses the moment bound to control the tail probability.
• Demonstrates that the improvement is non-trivial by showing that replacing ln(2n) with ln(2√n) reduces the failure probability δ by a factor of √n for fixed n.

Experimental results

Research questions

• RQ1 Can the logarithmic term ln(2n) in the PAC-Bayesian bound be reduced without increasing the KL divergence penalty?
• RQ2 What is the optimal order of dependence on n in the moment generating function of the KL divergence for [0,1]-valued random variables?
• RQ3 Is the √n term in the upper bound tight, and can it be improved?
• RQ4 Can the improved bound be applied to derive a tighter generalization error bound in learning theory?

Key findings

• The upper bound E[exp(n KL(M(X), μ))] ≤ 2√n holds for all n ≥ 8 and all [0,1]-valued i.i.d. random variables with mean μ.
• The lower bound E[exp(n KL(M(X), μ))] ≥ √n holds for all n ≥ 2 when the variables are nontrivial Bernoulli(μ) with μ ∈ (0,1).
• For Bernoulli variables, the expectation E[exp(n KL(M(X), μ))] is independent of μ and depends only on n, confirming the bound’s tightness across all biases.
• The improved PAC-Bayesian bound replaces ln(2n) with ln(2√n), reducing the failure probability δ by a factor of √n for fixed n.
• For n = 10,000, the new bound achieves the same generalization error as the original with failure probability < 1/10,000 instead of 1/100.
• The improvement is significant in terms of confidence, and no known method eliminates the logarithmic term without increasing the KL penalty, which would weaken the bound in practice.