[Paper Review] Adaptivity of averaged stochastic gradient descent to local strong convexity for logistic regression

Francis Bach|arXiv (Cornell University)|Mar 25, 2013

Stochastic Gradient Optimization Techniques39 references65 citations

TL;DR

This paper establishes that averaged stochastic gradient descent (ASGD) with a constant step size of order $1/R^2 oot{N}$ adapts to local strong convexity in logistic regression without prior knowledge of the strong convexity parameter $\mu$. It proves a convergence rate of $O(R^2/\mu N)$ when $\mu > R^2/\sqrt{N}$, demonstrating adaptivity to unknown local curvature via self-concordance properties of the logistic loss.

ABSTRACT

In this paper, we consider supervised learning problems such as logistic regression and study the stochastic gradient method with averaging, in the usual stochastic approximation setting where observations are used only once. We show that after $N$ iterations, with a constant step-size proportional to $1/R^2 \\sqrt{N}$ where $N$ is the number of observations and $R$ is the maximum norm of the observations, the convergence rate is always of order $O(1/\\sqrt{N})$, and improves to $O(R^2 / \\mu N)$ where $\\mu$ is the lowest eigenvalue of the Hessian at the global optimum (when this eigenvalue is greater than $R^2/\\sqrt{N}$). Since $\\mu$ does not need to be known in advance, this shows that averaged stochastic gradient is adaptive to \\emph{unknown local} strong convexity of the objective function. Our proof relies on the generalized self-concordance properties of the logistic loss and thus extends to all generalized linear models with uniformly bounded features.

Motivation & Objective

• To analyze the convergence behavior of averaged stochastic gradient descent (ASGD) in logistic regression under a finite-horizon, constant step-size setting.
• To establish that ASGD adapts to local strong convexity—specifically, the smallest eigenvalue $\mu$ of the Hessian at the optimum—without requiring $\mu$ to be known in advance.
• To derive a convergence rate that improves from $O(1/\sqrt{N})$ to $O(R^2/\mu N)$ when local strong convexity is present, without introducing exponential factors.
• To extend the analysis beyond global strong convexity by leveraging generalized self-concordance of the logistic loss and bounded feature norms.

Proposed method

• Uses a constant step size proportional to $1/R^2\sqrt{N}$, where $R$ is the maximum feature norm and $N$ is the number of observations.
• Applies Polyak-Ruppert averaging to iterates of stochastic gradient descent to improve convergence stability and rate.
• Employs generalized self-concordance properties of the logistic loss to control higher-order moments and derive concentration bounds.
• Derives a bound on the expected squared error $\mathbb{E}\|\bar{\theta}_N - \theta_*\|^2$ using a combination of exponential tail bounds and integral estimates over time.
• Introduces a threshold condition based on $\mu\sqrt{N}/R \geq 500$ to ensure the improved rate, with fallback to standard rates otherwise.
• Uses a doubling trick argument to extend results from constant to decaying step sizes, though the main analysis focuses on constant steps.

Experimental results

Research questions

• RQ1 Can averaged stochastic gradient descent achieve improved convergence rates in logistic regression when local strong convexity is present, even without knowing the strong convexity parameter $\mu$?
• RQ2 Does the convergence rate of ASGD adapt to the local curvature of the logistic loss, as measured by the smallest eigenvalue $\mu$ of the Hessian at the optimum?
• RQ3 Can the analysis avoid exponential dependence on the range of the linear predictor (e.g., $e^U$) when bounding the convergence rate?
• RQ4 Is it possible to achieve a rate of $O(R^2/\mu N)$ for ASGD in logistic regression under a constant step size, without requiring global strong convexity?

Key findings

• The expected squared error of the averaged iterate satisfies $\mathbb{E}\|\bar{\theta}_N - \theta_*\|^2 \leq \frac{R^2}{N\mu^2}(6\alpha + 21)^4$ when $\mu\sqrt{N}/R \geq 500$, with $\alpha = R\|\theta_0 - \theta_*\|$.
• When $\mu > R^2/\sqrt{N}$, the convergence rate improves from $O(1/\sqrt{N})$ to $O(R^2/\mu N)$, demonstrating adaptivity to local strong convexity.
• The improved rate is achieved with a constant step size of order $1/R^2\sqrt{N}$, and the analysis avoids exponential factors such as $e^U$ that often appear in similar bounds.
• The result extends to all generalized linear models with uniformly bounded features due to the self-concordance of the logistic loss.
• The analysis holds for finite $N$ and constant step sizes, and can be extended to decaying step sizes via a doubling trick.