[Paper Review] A Limitation of the PAC-Bayes Framework

Roi Livni, Shay Moran|arXiv (Cornell University)|Jun 24, 2020

Machine Learning and Algorithms34 references4 citations

TL;DR

This paper demonstrates a fundamental limitation of the PAC-Bayes framework by showing that even though 1-dimensional threshold classifiers are efficiently PAC-learnable with O(log(1/δ)/ε) samples, no PAC-Bayes analysis can yield a non-vacuous generalization bound for any algorithm learning this class. The key result is that for any such algorithm, there exists a realizable distribution where the PAC-Bayes bound becomes arbitrarily large, regardless of prior choice or algorithm design.

ABSTRACT

PAC-Bayes is a useful framework for deriving generalization bounds which was introduced by McAllester ('98). This framework has the flexibility of deriving distribution- and algorithm-dependent bounds, which are often tighter than VC-related uniform convergence bounds. In this manuscript we present a limitation for the PAC-Bayes framework. We demonstrate an easy learning task that is not amenable to a PAC-Bayes analysis. Specifically, we consider the task of linear classification in 1D; it is well-known that this task is learnable using just $O(\log(1/δ)/ε)$ examples. On the other hand, we show that this fact can not be proved using a PAC-Bayes analysis: for any algorithm that learns 1-dimensional linear classifiers there exists a (realizable) distribution for which the PAC-Bayes bound is arbitrarily large.

Motivation & Objective

• To identify and demonstrate a fundamental limitation of the PAC-Bayes framework in analyzing learnable concept classes.
• To show that even a simple, efficiently learnable class—1D thresholds—cannot be analyzed via non-vacuous PAC-Bayes bounds.
• To establish that this limitation holds even when allowing arbitrary, potentially distribution-dependent priors and any learning algorithm.
• To highlight the disconnect between PAC-learnability and PAC-Bayes amenability, challenging assumptions about the framework's universality.
• To motivate further exploration of alternative frameworks for analyzing generalization in stable or implicit-bias-based learning algorithms.

Proposed method

• Constructs a realizable distribution over 1D threshold examples where any Gibbs-classifier output by a learning algorithm results in an unbounded PAC-Bayes bound.
• Uses a probabilistic argument based on the structure of intervals and odd-indexed points to bound the probability of error events.
• Applies a KL divergence lower bound technique inspired by Alon et al. (2019) on differentially private learning to derive a lower bound on the KL divergence between posterior and prior.
• Employs a sampling-based argument over a set of candidate thresholds to show that the expected KL divergence must grow with the domain size.
• Leverages the fact that consistent empirical risk minimizers (ERMs) for thresholds are sufficient for PAC learning, yet still fail under PAC-Bayes analysis.
• Demonstrates that the bound's divergence is Ω((q₂−q₁)²b/log b), showing it grows with the domain size and cannot be made uniformly small.

Experimental results

Research questions

• RQ1 Can the PAC-Bayes framework provide non-vacuous generalization bounds for the class of 1-dimensional threshold classifiers?
• RQ2 Is there a learning algorithm for 1D thresholds that can be certified via a non-vacuous PAC-Bayes bound, even with distribution-dependent priors?
• RQ3 Does the existence of a highly efficient PAC learning algorithm for 1D thresholds imply that PAC-Bayes can also certify its generalization?
• RQ4 What are the structural limitations of the PAC-Bayes framework in capturing the generalization behavior of consistent, low-complexity learners?
• RQ5 Can PAC-Bayes analysis be extended to capture fast-converging, distribution-dependent learning rates in stable or interpolating algorithms?

Key findings

• The class of 1-dimensional thresholds is PAC-learnable with sample complexity O(log(1/δ)/ε), independent of domain size.
• For any algorithm learning 1D thresholds, there exists a realizable distribution for which the PAC-Bayes bound is arbitrarily large.
• This limitation holds even when using any prior distribution, including those that depend on the data distribution.
• The KL divergence term in the PAC-Bayes bound grows as Ω((q₂−q₁)²b/log b), indicating unbounded divergence for large domains.
• The result implies that no PAC-Bayes generalization bound can match the performance of 1-Nearest-Neighbor or other consistent learners on certain realizable distributions.
• The framework fails to capture the generalization of interpolating algorithms like 1-Nearest-Neighbor for this class, despite their strong empirical performance.