[論文レビュー] Data-Dependent Stability of Stochastic Gradient Descent

Ilja Kuzborskij, Christoph H. Lampert|arXiv (Cornell University)|Mar 5, 2017

Stochastic Gradient Optimization Techniques被引用数 64

ひとこと要約

この論文は、SGDに対してデータ依存の安定性の概念を導入し、初期化とデータ分布に依存する一般化境界を凸損失と非凸損失の両方で導出する。

ABSTRACT

We establish a data-dependent notion of algorithmic stability for Stochastic Gradient Descent (SGD), and employ it to develop novel generalization bounds. This is in contrast to previous distribution-free algorithmic stability results for SGD which depend on the worst-case constants. By virtue of the data-dependent argument, our bounds provide new insights into learning with SGD on convex and non-convex problems. In the convex case, we show that the bound on the generalization error depends on the risk at the initialization point. In the non-convex case, we prove that the expected curvature of the objective function around the initialization point has crucial influence on the generalization error. In both cases, our results suggest a simple data-driven strategy to stabilize SGD by pre-screening its initialization. As a corollary, our results allow us to show optimistic generalization bounds that exhibit fast convergence rates for SGD subject to a vanishing empirical risk and low noise of stochastic gradient.

研究の動機と目的

• Motivate and formalize a data-dependent stability notion for SGD beyond worst-case analyses.
• Derive generalization bounds for SGD in convex and non-convex settings that depend on initialization and data distribution.
• Show that stability improves when starting from low-risk, less-curved regions of the objective.
• Demonstrate optimistic bounds and transfer-learning implications using the data-dependent framework.

提案手法

• Define on-average stability that depends on algorithm parameters and data distribution (epsilon(theta)).
• Derive Theorem 3: convex losses with step sizes alpha_t ~ c/sqrt(t) yield epsilon(D, w1) bound involving initialization risk and gradient noise.
• Derive Theorem 4: non-convex losses with Lipschitz Hessian and step sizes alpha_t ~ c/t yield epsilon(D, w1) bound that incorporates curvature and initialization risk.
• Provide corollaries showing optimistic generalization rates and transfer-learning guidance.
• Present empirical validation comparing data-dependent bounds to worst-case bounds on a neural net example.
• Discuss an HTL (Hypothesis Transfer Learning) application where source hypotheses serve as initialization.
• Suggest a practical scheme to select favorable initializations to improve stability and transfer outcomes.

実験結果

リサーチクエスチョン

• RQ1 How can SGD generalization be bounded with a data-dependent stability notion rather than a distribution-free one?
• RQ2 How do initialization risk and local curvature influence SGD stability and generalization in convex and non-convex settings?
• RQ3 Can data-dependent stability lead to optimistic fast rates and inform transfer learning of SGD initialization?
• RQ4 How do transfer-learning scenarios affect stability bounds when source hypotheses initialize SGD?

主な発見

• SGD stability bounds depend on initialization risk and gradient noise in convex settings.
• In non-convex settings, the initialization curvature (second-order information) critically influences generalization bounds.
• Data-dependent bounds are tighter than distribution-free bounds in empirical tests on non-convex problems.
• Optimistic generalization bounds with fast rates are possible when empirical risk vanishes.
• A principled transfer-learning approach uses initialization from source hypotheses to minimize stability bounds.
• Evidence suggests SGD is more stable in less curved regions, aligning with observed deep-learning behavior.