[Paper Review] Efron-Stein PAC-Bayesian Inequalities

Ilja Kuzborskij, Csaba Szepesvári|arXiv (Cornell University)|Sep 4, 2019

Machine Learning and Algorithms43 references4 citations

TL;DR

This paper introduces semi-empirical Efron-Stein PAC-Bayesian inequalities that provide data-dependent concentration bounds for functions of independent random variables without requiring moment assumptions. By leveraging the Efron-Stein variance proxy and moment-generating functions, it derives novel Bernstein-type inequalities and applies them to unbounded loss functions, off-policy evaluation with Weighted Importance Sampling, and PAC-Bayesian learning, yielding tighter, truncation-free generalization bounds.

ABSTRACT

We prove semi-empirical concentration inequalities for random variables which are given as possibly nonlinear functions of independent random variables. These inequalities describe concentration of random variable in terms of the data/distribution-dependent Efron-Stein (ES) estimate of its variance and they do not require any additional assumptions on the moments. In particular, this allows us to state semi-empirical Bernstein type inequalities for general functions of unbounded random variables, which gives user-friendly concentration bounds for cases where related methods (e.g. bounded differences) might be more challenging to apply. We extend these results to Efron-Stein PAC-Bayesian inequalities which hold for arbitrary probability kernels that define a random, data-dependent choice of the function of interest. Finally, we demonstrate a number of applications, including PAC-Bayesian generalization bounds for unbounded loss functions, empirical Bernstein type generalization bounds, new truncation-free bounds for off-policy evaluation with Weighted Importance Sampling (WIS), and off-policy PAC-Bayesian learning with WIS.

Motivation & Objective

• To develop concentration inequalities for general functions of independent random variables that do not rely on moment assumptions or bounded differences.
• To extend the Efron-Stein variance proxy to PAC-Bayesian settings with data-dependent function selection.
• To derive semi-empirical Bernstein-type inequalities for unbounded random variables using the Efron-Stein estimate of variance.
• To enable tighter generalization bounds for learning with unbounded loss functions and off-policy evaluation using Weighted Importance Sampling.
• To provide truncation-free bounds in off-policy evaluation by replacing traditional truncation with moment-based control via Efron-Stein variance.

Proposed method

• Proposes a semi-empirical concentration inequality using the Efron-Stein variance proxy $ V^\text{{es}} $ as a data-dependent measure of sensitivity to coordinatewise perturbations.
• Derives an exponential Efron-Stein inequality relating the MGF of $ \Delta = f(S) - \mathbb{E}[f(S)] $ to the MGF of $ V^\text{{es}} $, enabling moment-based tail bounds.
• Applies the inequality to PAC-Bayesian settings by modeling the function choice as a random kernel $ \hat{p}_S $ over parameters $ \theta $, with prior $ p^0 $.
• Uses Gaussian integration and Cauchy-Schwarz to derive bounds on the MGF of $ \mathbb{E}[\Delta_\theta \mid S] $, leading to canonical pair properties for $ (\Delta_\theta, \sqrt{V_\theta}) $.
• Establishes a key inequality (Eq. 13) bounding the MGF of $ \mathbb{E}[\Delta_\theta \mid S] $ in terms of $ \mathrm{KL}(\hat{p}_S \| p^0) $ and $ \mathbb{E}[V_\theta \mid S] $.
• Applies Lemma 3 on sub-Gaussian tails to derive a second bound (Eq. 14), ensuring sub-Gaussian behavior under moment control.

Experimental results

Research questions

• RQ1 Can we derive tighter concentration bounds for unbounded functions without assuming bounded differences or moment conditions?
• RQ2 How can the Efron-Stein variance proxy be used to construct data-dependent, semi-empirical Bernstein-type inequalities?
• RQ3 Can Efron-Stein-based inequalities be extended to PAC-Bayesian frameworks with random, data-dependent function selection?
• RQ4 Can these inequalities yield truncation-free bounds for off-policy evaluation using Weighted Importance Sampling?
• RQ5 Can the resulting bounds be applied to generalization in learning with unbounded loss functions?

Key findings

• The paper establishes a semi-empirical Bernstein-type inequality that controls the tail of $ \Delta $ using the first and second moments of the Efron-Stein variance proxy, avoiding assumptions on higher-order moments.
• It proves that for weakly self-bounding functions satisfying $ V^\text{{es}} \leq a f(S) + b $, the bound $ \mathbb{P}(\Delta \leq 2\sqrt{(a\mathbb{E}[f(S)] + b)x} + 2ax) \geq 1 - e^{-x} $ holds for all $ x \geq 0 $.
• The PAC-Bayesian Efron-Stein inequality (Eq. 13) provides a bound on the MGF of $ \mathbb{E}[\Delta_\theta \mid S] $ in terms of the KL divergence between posterior and prior, and the conditional variance proxy $ \mathbb{E}[V_\theta \mid S] $.
• A second bound (Eq. 14) is derived using Lemma 3, showing sub-Gaussian behavior under moment control, with $ \mathbb{E}[\exp(U^2/4)] \leq 2 $, leading to $ \mathbb{E}[\exp(xU)] \leq 2e^{x^2} $.
• The framework yields truncation-free generalization bounds for off-policy evaluation with Weighted Importance Sampling, replacing traditional truncation with moment-based control.
• The results are applied to RLS, ERM, and SGLD in domain adaptation, demonstrating tighter bounds for unbounded losses and improved generalization in off-policy learning.