[Paper Review] Adaptive SGD with Polyak stepsize and Line-search: Robust Convergence and Variance Reduction

Xiaowen Jiang, Sebastian U. Stich|arXiv (Cornell University)|Aug 11, 2023

Domain Adaptation and Few-Shot LearningComputer Science3 citations

TL;DR

This paper proposes AdaSPS and AdaSLS, adaptive stochastic gradient descent variants that achieve robust convergence in both interpolation and non-interpolation settings without requiring problem-specific parameters. By integrating a novel variance reduction technique, the algorithms attain a gradient complexity of $\widetilde{\mathcal{O}}(n + 1/\varepsilon)$ for $\mathcal{O}(\varepsilon)$-suboptimality in convex functions, matching fast rates of AdaSVRG while avoiding inner-outer loop structures.

ABSTRACT

The recently proposed stochastic Polyak stepsize (SPS) and stochastic line-search (SLS) for SGD have shown remarkable effectiveness when training over-parameterized models. However, in non-interpolation settings, both algorithms only guarantee convergence to a neighborhood of a solution which may result in a worse output than the initial guess. While artificially decreasing the adaptive stepsize has been proposed to address this issue (Orvieto et al. [2022]), this approach results in slower convergence rates for convex and over-parameterized models. In this work, we make two contributions: Firstly, we propose two new variants of SPS and SLS, called AdaSPS and AdaSLS, which guarantee convergence in non-interpolation settings and maintain sub-linear and linear convergence rates for convex and strongly convex functions when training over-parameterized models. AdaSLS requires no knowledge of problem-dependent parameters, and AdaSPS requires only a lower bound of the optimal function value as input. Secondly, we equip AdaSPS and AdaSLS with a novel variance reduction technique and obtain algorithms that require $\smash{\widetilde{\mathcal{O}}}(n+1/ε)$ gradient evaluations to achieve an $\mathcal{O}(ε)$-suboptimality for convex functions, which improves upon the slower $\mathcal{O}(1/ε^2)$ rates of AdaSPS and AdaSLS without variance reduction in the non-interpolation regimes. Moreover, our result matches the fast rates of AdaSVRG but removes the inner-outer-loop structure, which is easier to implement and analyze. Finally, numerical experiments on synthetic and real datasets validate our theory and demonstrate the effectiveness and robustness of our algorithms.

Motivation & Objective

• To address the lack of robust convergence in existing adaptive SGD methods like SPS and SLS when training in non-interpolation settings.
• To develop adaptive stepsize methods that maintain fast convergence rates in both convex and strongly convex regimes without requiring knowledge of problem-dependent parameters.
• To integrate variance reduction with Polyak and line-search type stepsizes while preserving theoretical guarantees in non-interpolation regimes.
• To eliminate the need for inner-outer loop structures common in variance-reduced methods like AdaSVRG, improving implementability and analysis.
• To validate the theoretical improvements through numerical experiments on synthetic and real-world datasets.

Proposed method

• Proposes AdaSPS, a variant of stochastic Polyak stepsize that uses a lower bound on the optimal function value to ensure convergence in non-interpolation settings.
• Introduces AdaSLS, a line-search-based adaptive method requiring no problem-dependent parameters, enabling robust convergence across settings.
• Employs a novel variance reduction mechanism that dynamically adjusts the stepsize using cumulative function value differences and gradient norms.
• Derives convergence rates by analyzing the interplay between adaptive stepsize selection and variance reduction, ensuring sub-linear and linear convergence for convex and strongly convex functions.
• Designs a restart mechanism for deep learning extensions, allowing stepsize increases to mimic cyclic behavior observed in practice.
• Uses a hybrid update rule that combines adaptive stepsize with backtracking to maintain stability and convergence.

Experimental results

Research questions

• RQ1 Can adaptive SGD with Polyak stepsize and line-search achieve robust convergence in both interpolation and non-interpolation settings?
• RQ2 Can variance reduction be successfully integrated with Polyak and line-search stepsizes while preserving theoretical convergence guarantees?
• RQ3 Do the proposed algorithms achieve faster convergence rates than existing adaptive methods in non-interpolation regimes?
• RQ4 Can the algorithms match the fast rates of AdaSVRG without using an inner-outer loop structure?
• RQ5 Is the proposed method effective and robust in non-convex deep learning settings?

Key findings

• AdaSPS and AdaSLS guarantee convergence in non-interpolation settings, overcoming the limitation of SPS and SLS which only converge to a neighborhood of the solution.
• AdaSPS requires only a lower bound on the optimal function value, while AdaSLS requires no problem-dependent parameters, enhancing practical usability.
• The variance-reduced variants achieve a gradient complexity of $\widetilde{\mathcal{O}}(n + 1/\varepsilon)$ for $\mathcal{O}(\varepsilon)$-suboptimality in convex functions, improving upon the $\mathcal{O}(1/\varepsilon^2)$ rate of non-variance-reduced versions.
• The proposed algorithms match the fast convergence rates of AdaSVRG but avoid the complex inner-outer loop structure, simplifying implementation and analysis.
• Numerical experiments on synthetic and real datasets confirm the theoretical findings and demonstrate superior robustness and performance compared to SPS, DecSPS, Adam, and SGD with momentum.
• The non-convex extension, AdaSPS (DL), shows competitive performance on CIFAR10 and CIFAR100 using ResNet-34, suggesting practical promise beyond convex settings.