[Paper Review] Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)

Francis Bach, Éric Moulines|arXiv (Cornell University)|Jun 10, 2013

Stochastic Gradient Optimization Techniques34 references228 citations

TL;DR

This paper proposes two novel stochastic optimization algorithms that achieve an O(1/n) convergence rate for non-strongly convex, smooth problems—unlike standard stochastic gradient descent, which only achieves O(1/√n). For least-squares regression, it shows that averaged stochastic gradient descent with constant step-size attains O(1/n) convergence; for logistic regression, it introduces a new algorithm that builds local quadratic approximations while maintaining SGD's computational efficiency.

ABSTRACT

We consider the stochastic approximation problem where a convex function has to be minimized, given only the knowledge of unbiased estimates of its gradients at certain points, a framework which includes machine learning methods based on the minimization of the empirical risk. We focus on problems without strong convexity, for which all previously known algorithms achieve a convergence rate for function values of O(1/n^{1/2}). We consider and analyze two algorithms that achieve a rate of O(1/n) for classical supervised learning problems. For least-squares regression, we show that averaged stochastic gradient descent with constant step-size achieves the desired rate. For logistic regression, this is achieved by a simple novel stochastic gradient algorithm that (a) constructs successive local quadratic approximations of the loss functions, while (b) preserving the same running time complexity as stochastic gradient descent. For these algorithms, we provide a non-asymptotic analysis of the generalization error (in expectation, and also in high probability for least-squares), and run extensive experiments on standard machine learning benchmarks showing that they often outperform existing approaches.

Motivation & Objective

• Address the gap in convergence rates for non-strongly convex stochastic optimization, where standard methods achieve only O(1/√n).
• Develop algorithms that achieve O(1/n) convergence without requiring strong convexity, which is often absent in high-dimensional machine learning problems.
• Preserve computational efficiency comparable to standard stochastic gradient descent while improving convergence speed.
• Provide non-asymptotic generalization error bounds in expectation and high probability for the proposed algorithms.
• Demonstrate empirically that the proposed methods outperform existing approaches on standard machine learning benchmarks.

Proposed method

• For least-squares regression, analyze averaged stochastic gradient descent with constant step-size, proving it achieves O(1/n) convergence under smoothness and moment conditions.
• For logistic regression, propose a new stochastic algorithm that constructs successive local quadratic approximations of the loss function.
• Ensure the new algorithm maintains the same per-iteration complexity as standard SGD, enabling scalability to large datasets.
• Use non-asymptotic analysis to bound the expected and high-probability generalization error for both algorithms.
• Leverage smoothness of the loss functions (square and logistic) to derive tighter convergence rates despite lack of strong convexity.
• Introduce a novel analysis framework involving operator inequalities and moment bounds to control the evolution of the iterates and their deviation from the optimal solution.

Experimental results

Research questions

• RQ1 Can stochastic optimization achieve O(1/n) convergence for non-strongly convex problems, given that standard methods are limited to O(1/√n)?
• RQ2 Does constant-step-size averaged SGD achieve O(1/n) convergence in least-squares regression without strong convexity?
• RQ3 Can a stochastic algorithm for logistic regression achieve O(1/n) convergence by combining local quadratic approximations with low computational overhead?
• RQ4 What are the non-asymptotic generalization error bounds for the proposed algorithms in expectation and high probability?
• RQ5 How do the proposed algorithms compare empirically to existing methods on standard machine learning benchmarks?

Key findings

• Averaged stochastic gradient descent with constant step-size achieves O(1/n) convergence rate for least-squares regression without requiring strong convexity.
• For logistic regression, the proposed algorithm with local quadratic approximations achieves O(1/n) convergence while preserving the same running time complexity as standard SGD.
• The generalization error for the least-squares algorithm is bounded in expectation and in high probability, with explicit dependence on problem parameters.
• Empirical results on datasets including quantum, rcv1, and news show the proposed methods often outperform existing approaches in terms of test performance and training objective convergence.
• On non-sparse datasets, SAG achieves the smallest training error, but on high-dimensional sparse datasets, constant-step-size SGD performs best, confirming the practical advantage of the proposed method.
• Theoretical analysis confirms that the convergence rate improvement is due to exploiting smoothness of the loss functions, even when strong convexity is absent.