[논문 리뷰] High probability generalization bounds for uniformly stable algorithms with nearly optimal rate

Vitaly Feldman, J. Vondrák|arXiv (Cornell University)|2019. 02. 27.

Stochastic Gradient Optimization Techniques참고 문헌 44인용 수 40

한 줄 요약

본 논문은 gamma-uniformly stable 알고리즘에 대해 로그 오버헤드를 가지는 거의 타이트한 고확률 일반화 경계를 보이고, 기존 경계보다 개선하며 SGD 및 정규화된 ERM에 대한 강력한 결과를 가능하게 한다.

ABSTRACT

Algorithmic stability is a classical approach to understanding and analysis of the generalization error of learning algorithms. A notable weakness of most stability-based generalization bounds is that they hold only in expectation. Generalization with high probability has been established in a landmark paper of Bousquet and Elisseeff (2002) albeit at the expense of an additional $\sqrt{n}$ factor in the bound. Specifically, their bound on the estimation error of any $γ$-uniformly stable learning algorithm on $n$ samples and range in $[0,1]$ is $O(γ\sqrt{n \log(1/δ)} + \sqrt{\log(1/δ)/n})$ with probability $\geq 1-δ$. The $\sqrt{n}$ overhead makes the bound vacuous in the common settings where $γ\geq 1/\sqrt{n}$. A stronger bound was recently proved by the authors (Feldman and Vondrak, 2018) that reduces the overhead to at most $O(n^{1/4})$. Still, both of these results give optimal generalization bounds only when $γ= O(1/n)$. We prove a nearly tight bound of $O(γ\log(n)\log(n/δ) + \sqrt{\log(1/δ)/n})$ on the estimation error of any $γ$-uniformly stable algorithm. It implies that for algorithms that are uniformly stable with $γ= O(1/\sqrt{n})$, estimation error is essentially the same as the sampling error. Our result leads to the first high-probability generalization bounds for multi-pass stochastic gradient descent and regularized ERM for stochastic convex problems with nearly optimal rate --- resolving open problems in prior work. Our proof technique is new and we introduce several analysis tools that might find additional applications.

연구 동기 및 목표

• Motivate and analyze high-probability generalization bounds for uniformly stable learning algorithms.
• Improve over previous bounds by reducing overhead from sqrt(n) to polylog factors in delta and n.
• Show applicability to multi-pass SGD and regularized ERM in stochastic convex settings.
• Provide new proof techniques that decompose estimation error via range reduction and dataset partitioning.

제안 방법

• Define gamma-uniform stability for data-dependent functions and relate to generalization error Delta_s(M).
• Introduce a leave-one-out unbiased transformation L(s,z) with controlled stability.
• Develop range reduction by clamping and mean subtraction while preserving zero-mean and stability.
• Use a recursive two-operation scheme (range reduction and dataset size reduction) to bound estimation error.
• Prove a main concentration bound with exponential tails: Delta_s(M) <= c( gamma log(n) log(n/delta) + sqrt(log(1/delta)/n) ).
• Apply the bound to stochastic convex optimization settings and derive corollaries for strongly convex ERM and SGD.]

실험 결과

연구 질문

• RQ1 Can high-probability generalization bounds be achieved for gamma-uniformly stable algorithms with only logarithmic overhead?
• RQ2 How does the stability parameter gamma interact with logarithmic factors to affect estimation error in practice?
• RQ3 Do the new bounds yield meaningful guarantees for multi-pass SGD and regularized ERM in stochastic convex problems?
• RQ4 Can new range-reduction (clamping) techniques preserve unbiasedness and stability simultaneously?
• RQ5 How can the results inform prediction-private learning and related privacy-preserving settings?

주요 결과

• Established a high-probability bound: Pr[Delta_s(M) >= c( gamma log(n) log(n/delta) + sqrt(log(1/delta)/n) )] <= delta.
• Showed that for gamma = O(1/√n), estimation error matches sampling error up to logarithmic factors.
• Derived nearly optimal high-probability generalization bounds for full gradient descent in SGD and for regularized ERM in stochastic convex problems.
• Provided corollaries giving high-probability guarantees with regularization parameter lambda = log(n)/√n.
• Demonstrated applicability to differential privacy contexts via prediction private learning and related bounds.