[Paper Review] Accelerated, Optimal, and Parallel: Some Results on Model-Based Stochastic Optimization

Karan N. Chadha, Gary Cheng+1 more|arXiv (Cornell University)|Jan 7, 2021

Stochastic Gradient Optimization Techniques45 references5 citations

TL;DR

This paper introduces accelerated, optimal, and parallel model-based stochastic optimization methods that achieve order-optimal convergence rates with linear speedup in minibatch size. By extending the aProx framework to minibatching and acceleration, the authors provide non-asymptotic guarantees and establish new fundamental constants for interpolation problems, demonstrating improved robustness and scalability over standard stochastic gradient methods.

ABSTRACT

We extend the Approximate-Proximal Point (aProx) family of model-based methods for solving stochastic convex optimization problems, including stochastic subgradient, proximal point, and bundle methods, to the minibatch and accelerated setting. To do so, we propose specific model-based algorithms and an acceleration scheme for which we provide non-asymptotic convergence guarantees, which are order-optimal in all problem-dependent constants and provide linear speedup in minibatch size, while maintaining the desirable robustness traits (e.g. to stepsize) of the aProx family. Additionally, we show improved convergence rates and matching lower bounds identifying new fundamental constants for "interpolation" problems, whose importance in statistical machine learning is growing; this, for example, gives a parallelization strategy for alternating projections. We corroborate our theoretical results with empirical testing to demonstrate the gains accurate modeling, acceleration, and minibatching provide.

Motivation & Objective

• To develop parallelizable and minibatch-aware model-based optimization methods that overcome the sequential nature of existing aProx methods.
• To provide non-asymptotic convergence guarantees that depend on variance of gradient estimates, enabling linear speedup with minibatch size.
• To identify and analyze new fundamental problem constants for interpolation problems, which are increasingly important in modern machine learning.
• To design an accelerated variant of the aProx method with optimal convergence rates and robustness to hyperparameter tuning.
• To empirically validate the theoretical gains in robustness, acceleration, and model accuracy through comprehensive experiments.

Proposed method

• Extends the aProx framework to minibatched settings by using models of the objective function $ F(x; s) $ that satisfy convexity and local accuracy conditions.
• Proposes a new acceleration scheme based on Nesterov-style momentum, using a three-point update rule involving $ x_k, y_k, z_k $, and a sequence of stepsize parameters $ \theta_k, \alpha_k, \eta_k $.
• Derives a single-step progress bound using Fenchel-Young inequality and strong convexity of the Bregman divergence $ D_h $, leading to a recursive error bound.
• Introduces a novel error decomposition $ \zeta_k = e_k + \langle \xi_k, z_k - y_k \rangle + \frac{\|\xi_k\|_*^2 - \sigma_0^2}{2\alpha_k} $ to handle model and stochastic errors.
• Uses a recursive summation of the error bound and applies telescoping sums over Bregman divergences to derive convergence rates.
• Employs a specific choice of $ \theta_k = \frac{2}{k+2} $, $ \eta_k = \eta_0 \frac{2\sqrt{k}}{k+2} $, and $ \alpha_k = \frac{1}{L\theta_k + \eta_k} $ to achieve optimal convergence.

Experimental results

Research questions

• RQ1 Can the aProx family of model-based methods be extended to support minibatching while maintaining convergence guarantees and robustness?
• RQ2 What is the optimal convergence rate for model-based stochastic optimization under variance-dependent error bounds?
• RQ3 How does acceleration affect the convergence of model-based methods, and can it be made order-optimal?
• RQ4 What new fundamental constants govern convergence in interpolation problems, and how do they affect parallelization efficiency?
• RQ5 Can improved model accuracy and minibatching lead to practical gains in robustness and speed?

Key findings

• The proposed method achieves linear speedup in minibatch size, reducing convergence rate from $ 1/\sqrt{k} $ to $ 1/\sqrt{km} $, matching standard gradient methods.
• The accelerated aProx variant achieves order-optimal convergence rates, with $ \mathbb{E}[f(x_k)+r(x_k)-f(x^*)-r(x^*)] \leq \mathcal{O}(1/k^2) $ under appropriate parameter choices.
• For interpolation problems, the paper identifies a new growth condition as fundamental to convergence difficulty, and provides matching lower bounds for this class.
• The method demonstrates improved robustness to stepsize selection and noise, outperforming standard stochastic gradient methods in empirical evaluations.
• Theoretical analysis confirms that the error bound scales as $ \mathcal{O}(1/\sqrt{k}) $ with $ \sigma^2 $-bounded noise, and the acceleration scheme achieves $ \mathcal{O}(1/k^2) $ convergence.
• Empirical results show that accurate modeling and minibatching significantly improve convergence speed and stability, especially in ill-conditioned or noisy settings.