[Paper Review] High probability generalization bounds for uniformly stable algorithms with nearly optimal rate

Vitaly Feldman, J. Vondrák|arXiv (Cornell University)|Feb 27, 2019

Stochastic Gradient Optimization Techniques44 references40 citations

TL;DR

The paper proves a nearly tight high-probability generalization bound for gamma-uniformly stable algorithms with a logarithmic overhead, improving over prior bounds and enabling strong results for SGD and regularized ERM.

ABSTRACT

Algorithmic stability is a classical approach to understanding and analysis of the generalization error of learning algorithms. A notable weakness of most stability-based generalization bounds is that they hold only in expectation. Generalization with high probability has been established in a landmark paper of Bousquet and Elisseeff (2002) albeit at the expense of an additional $\sqrt{n}$ factor in the bound. Specifically, their bound on the estimation error of any $γ$-uniformly stable learning algorithm on $n$ samples and range in $[0,1]$ is $O(γ\sqrt{n \log(1/δ)} + \sqrt{\log(1/δ)/n})$ with probability $\geq 1-δ$. The $\sqrt{n}$ overhead makes the bound vacuous in the common settings where $γ\geq 1/\sqrt{n}$. A stronger bound was recently proved by the authors (Feldman and Vondrak, 2018) that reduces the overhead to at most $O(n^{1/4})$. Still, both of these results give optimal generalization bounds only when $γ= O(1/n)$. We prove a nearly tight bound of $O(γ\log(n)\log(n/δ) + \sqrt{\log(1/δ)/n})$ on the estimation error of any $γ$-uniformly stable algorithm. It implies that for algorithms that are uniformly stable with $γ= O(1/\sqrt{n})$, estimation error is essentially the same as the sampling error. Our result leads to the first high-probability generalization bounds for multi-pass stochastic gradient descent and regularized ERM for stochastic convex problems with nearly optimal rate --- resolving open problems in prior work. Our proof technique is new and we introduce several analysis tools that might find additional applications.

Motivation & Objective

• Motivate and analyze high-probability generalization bounds for uniformly stable learning algorithms.
• Improve over previous bounds by reducing overhead from sqrt(n) to polylog factors in delta and n.
• Show applicability to multi-pass SGD and regularized ERM in stochastic convex settings.
• Provide new proof techniques that decompose estimation error via range reduction and dataset partitioning.

Proposed method

• Define gamma-uniform stability for data-dependent functions and relate to generalization error Delta_s(M).
• Introduce a leave-one-out unbiased transformation L(s,z) with controlled stability.
• Develop range reduction by clamping and mean subtraction while preserving zero-mean and stability.
• Use a recursive two-operation scheme (range reduction and dataset size reduction) to bound estimation error.
• Prove a main concentration bound with exponential tails: Delta_s(M) <= c( gamma log(n) log(n/delta) + sqrt(log(1/delta)/n) ).
• Apply the bound to stochastic convex optimization settings and derive corollaries for strongly convex ERM and SGD.

Experimental results

Research questions

• RQ1 Can high-probability generalization bounds be achieved for gamma-uniformly stable algorithms with only logarithmic overhead?
• RQ2 How does the stability parameter gamma interact with logarithmic factors to affect estimation error in practice?
• RQ3 Do the new bounds yield meaningful guarantees for multi-pass SGD and regularized ERM in stochastic convex problems?
• RQ4 Can new range-reduction (clamping) techniques preserve unbiasedness and stability simultaneously?
• RQ5 How can the results inform prediction-private learning and related privacy-preserving settings?

Key findings

• Established a high-probability bound: Pr[Delta_s(M) >= c( gamma log(n) log(n/delta) + sqrt(log(1/delta)/n) )] <= delta.
• Showed that for gamma = O(1/√n), estimation error matches sampling error up to logarithmic factors.
• Derived nearly optimal high-probability generalization bounds for full gradient descent in SGD and for regularized ERM in stochastic convex problems.
• Provided corollaries giving high-probability guarantees with regularization parameter lambda = log(n)/√n.
• Demonstrated applicability to differential privacy contexts via prediction private learning and related bounds.