[Paper Review] A simpler approach to obtaining an O(1/t) convergence rate for the projected stochastic subgradient method

Simon Lacoste-Julien, Mark Schmidt+1 more|arXiv (Cornell University)|Dec 10, 2012

Stochastic Gradient Optimization TechniquesComputer Science10 references152 citations

TL;DR

This paper presents a simplified weighted averaging technique for the projected stochastic subgradient method that achieves an O(1/t) convergence rate for strongly convex functions. By using a time-varying weight of (t+1) for each iterate w_t and updating the average with ρ_t = 2/(t+2), the method ensures optimal O(1/t) convergence with a significantly simpler proof and implementation compared to prior approaches.

ABSTRACT

In this note, we present a new averaging technique for the projected stochastic subgradient method. By using a weighted average with a weight of t+1 for each iterate w_t at iteration t, we obtain the convergence rate of O(1/t) with both an easy proof and an easy implementation. The new scheme is compared empirically to existing techniques, with similar performance behavior.

Motivation & Objective

• To simplify the analysis and implementation of the projected stochastic subgradient method while maintaining optimal O(1/t) convergence for strongly convex objectives.
• To replace complex averaging schemes with a weighted average that assigns increasing weight (t+1) to each iterate w_t at time t.
• To achieve a tighter convergence rate with a more straightforward proof than existing methods, particularly for non-smooth, strongly convex problems.
• To demonstrate that the new scheme matches or outperforms standard averaging in empirical performance while simplifying online implementation.

Proposed method

• The method uses a weighted average of iterates w_t, defined as w̄_T = (2/(T+1)(T+2)) * Σ_{t=0}^{T} (t+1) w_t, to estimate the minimizer.
• The averaging is updated online via w̄_t = (1 - ρ_t) w̄_{t-1} + ρ_t w_t with ρ_t = 2/(t+2), enabling efficient online computation.
• A step size of γ_t = 2/(μ(t+1)) is used, which is larger than the classical 1/(μt) and improves convergence rate.
• The convergence proof leverages a telescoping sum over t * [𝔼f(w_{t-1}) - f(w*)], using the strong convexity and finite variance assumptions.
• The analysis avoids complex martingale or concentration arguments, relying instead on elementary inequalities and Minkowski’s inequality for norm expectations.
• The method is validated empirically on SVM-like problems, showing comparable performance to existing schemes with simpler implementation.

Experimental results

Research questions

• RQ1 Can a simpler averaging scheme achieve O(1/t) convergence for the projected stochastic subgradient method in the strongly convex, non-smooth case?
• RQ2 Does a weighted average with time-varying weights (t+1) for each iterate w_t yield a tighter and more easily provable convergence rate than uniform averaging?
• RQ3 Can the convergence proof be simplified without sacrificing the rate or constant tightness, especially for non-smooth objectives?
• RQ4 How does the proposed scheme compare empirically to standard averaging and other weighted schemes in terms of convergence speed and stability?

Key findings

• The proposed weighted average achieves an O(1/t) convergence rate for the expected objective gap, improving upon the classical O((log t)/t) rate of uniform averaging.
• The method ensures 𝔼‖w_T - w*‖² ≤ 4B²/(μ²(T+1)), which is O(1/T), with a tighter constant than prior approaches.
• The convergence rate is achieved with a simple online update rule using ρ_t = 2/(t+2), which is computationally efficient and easy to implement.
• The proof is significantly simpler than prior works, avoiding complex martingale or concentration bounds, and relies only on basic inequalities and telescoping sums.
• Empirical results show performance comparable to existing schemes, confirming that the simplification does not compromise practical effectiveness.
• The method is applicable to a wide range of problems, including SVM and structured prediction, under standard assumptions of subgradient unbiasedness and finite variance.