[Paper Review] Robustness Implies Generalization via Data-Dependent Generalization Bounds

Kenji Kawaguchi, Zhun Deng+2 more|arXiv (Cornell University)|Jun 27, 2022

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper establishes that robustness implies generalization through novel data-dependent generalization bounds that eliminate dependence on the hypothesis space and significantly reduce reliance on the covering number K. By introducing an improved concentration bound for multinomial random variables and leveraging random projections to construct adaptive partitions, the authors achieve near-exponential improvements in generalization bounds for lasso and deep learning models, validated on real-world data and theoretical models.

ABSTRACT

This paper proves that robustness implies generalization via data-dependent generalization bounds. As a result, robustness and generalization are shown to be connected closely in a data-dependent manner. Our bounds improve previous bounds in two directions, to solve an open problem that has seen little development since 2010. The first is to reduce the dependence on the covering number. The second is to remove the dependence on the hypothesis space. We present several examples, including ones for lasso and deep learning, in which our bounds are provably preferable. The experiments on real-world data and theoretical models demonstrate near-exponential improvements in various situations. To achieve these improvements, we do not require additional assumptions on the unknown distribution; instead, we only incorporate an observable and computable property of the training samples. A key technical innovation is an improved concentration bound for multinomial random variables that is of independent interest beyond robustness and generalization.

Motivation & Objective

• Address the long-standing open problem of reducing the square root dependence on the covering number K in algorithmic robustness bounds since 2010.
• Eliminate the dependence on the hypothesis space, which limits the applicability of prior generalization bounds.
• Develop a data-dependent framework that uses observable, computable properties of training samples to improve generalization guarantees.
• Provide provably tighter bounds for robustness-to-generalization connections in high-dimensional settings, especially for deep learning and lasso.
• Introduce a novel concentration bound for multinomial random variables that is of independent interest beyond generalization theory.

Proposed method

• Propose a data-dependent generalization bound that replaces the standard K-dependent term with a refined, adaptive partitioning scheme based on random projections.
• Use random matrix projection to map high-dimensional input space into a lower-dimensional space, then apply ε-covering on the projected space to define pre-partitions.
• Define the final partition as the inverse image of the projected space's covering, reducing K compared to direct covering of the original space.
• Introduce a new concentration inequality for multinomial random variables that improves the dependence on K from √K to logarithmic or sub-polynomial terms.
• Apply the improved concentration bound to derive tighter generalization error bounds that scale with ε(S) and a reduced function of K, rather than √K.
• Validate the method using real-world datasets (e.g., Semeion, MNIST) and synthetic models, comparing K and generalization error across different data distributions.

Experimental results

Research questions

• RQ1 Can the dependence on the covering number K in algorithmic robustness-based generalization bounds be significantly reduced without additional assumptions on the data distribution?
• RQ2 Does removing the hypothesis space dependence lead to tighter and more practical generalization bounds in high-dimensional learning problems?
• RQ3 Can random projections be used to construct adaptive partitions that yield provably better generalization bounds than standard ε-coverings?
• RQ4 Is the proposed concentration bound for multinomial random variables superior to existing bounds in terms of K-dependence and applicability?
• RQ5 Do the new data-dependent bounds outperform prior bounds in real-world deep learning and lasso models?

Key findings

• The proposed generalization bounds achieve near-exponential improvements over prior bounds in terms of K-dependence, especially in high-dimensional settings.
• The bounds eliminate dependence on the hypothesis space, making them applicable to a broader class of models including deep neural networks and lasso.
• Experiments on real-world data and synthetic models show that the new bounds are provably tighter and more favorable than existing bounds.
• The use of random projections to define partitions reduces K significantly compared to direct covering of the input space, especially as dimension d increases.
• The improved concentration bound for multinomial variables reduces the K-dependence from √K to sub-polynomial or logarithmic terms, resolving a long-standing open problem.
• The method achieves tighter generalization bounds for lasso and deep learning models, with empirical validation showing consistent improvements across multiple datasets and distributions.