[Paper Review] Asynchronous Iterations in Optimization: New Sequence Results and Sharper Algorithmic Guarantees

Hamid Reza Feyzmahdavian, Mikael Johansson|arXiv (Cornell University)|Sep 9, 2021

Stochastic Gradient Optimization Techniques4 citations

TL;DR

This paper introduces novel sequence-based convergence results for asynchronous iterations in optimization, enabling sharper, explicit bounds on convergence rates that depend on asynchrony levels. By unifying analysis of key algorithmic families, it improves iteration complexity for proximal incremental gradient methods, tightens guarantees for asynchronous SGD using average delay, and provides tighter convergence rates for block-coordinate and totally asynchronous schemes under general delay models.

ABSTRACT

We introduce novel convergence results for asynchronous iterations that appear in the analysis of parallel and distributed optimization algorithms. The results are simple to apply and give explicit estimates for how the degree of asynchrony impacts the convergence rates of the iterates. Our results shorten, streamline and strengthen existing convergence proofs for several asynchronous optimization methods and allow us to establish convergence guarantees for popular algorithms that were thus far lacking a complete theoretical understanding. Specifically, we use our results to derive better iteration complexity bounds for proximal incremental aggregated gradient methods, to obtain tighter guarantees depending on the average rather than maximum delay for the asynchronous stochastic gradient descent method, to provide less conservative analyses of the speedup conditions for asynchronous block-coordinate implementations of Krasnoselskii-Mann iterations, and to quantify the convergence rates for totally asynchronous iterations under various assumptions on communication delays and update rates.

Motivation & Objective

• To address the lack of explicit, tight convergence rate guarantees for asynchronous optimization algorithms under realistic asynchrony models.
• To unify and streamline convergence proofs for popular asynchronous methods like Hogwild!, ASGD, ARock, and Asaga, which previously relied on complex, ad hoc inductions.
• To develop generalizable theoretical tools that capture the structural impact of asynchrony rather than treating it as noise or worst-case perturbation.
• To quantify convergence rates under both bounded and unbounded communication delays, particularly for totally asynchronous iterations.
• To improve algorithmic efficiency by enabling larger step-sizes and tighter complexity bounds depending on average rather than maximum delay.

Proposed method

• Introduces two families of sequences defined by specific inequalities that model the dynamics of asynchronous iterations in optimization.
• Derives general convergence theorems for these sequence families, providing explicit decay rates based on asynchrony parameters such as maximum or average delay.
• Applies these sequence results to analyze proximal incremental aggregated gradient methods, deriving improved iteration complexity bounds.
• Uses the framework to analyze asynchronous stochastic gradient descent with delay-dependent step-sizes, yielding tighter bounds that depend on average delay instead of maximum delay.
• Applies the results to Krasnosel’ski’- Mann iterations under block-coordinate updates, improving speedup and step-size range guarantees.
• Establishes convergence rates for totally asynchronous iterations under unbounded, linearly growing delays by constructing Lyapunov-like functions with polynomial decay rates.

Experimental results

Research questions

• RQ1 How can we derive explicit, non-asymptotic convergence rate bounds for asynchronous optimization algorithms that depend on the degree of asynchrony?
• RQ2 Can we unify and simplify existing convergence proofs for asynchronous methods like ASGD, ARock, and proximal IAG by identifying common sequence structures?
• RQ3 To what extent can convergence guarantees be improved by using average delay instead of maximum delay in step-size rules and complexity bounds?
• RQ4 What convergence rates can be established for totally asynchronous iterations when communication delays grow unboundedly?
• RQ5 How do the proposed sequence results improve the theoretical understanding of block-coordinate and pseudo-contraction-based asynchronous algorithms?

Key findings

• For proximal incremental aggregated gradient methods on convex, L-smooth objectives, the iteration complexity is improved to O(Lτ/ε), a significant reduction from the prior bound of O(L²τ³/ε).
• Under a quadratic functional growth condition, the iteration complexity becomes O(Qτ log(1/ε)), where Q = L/μ is the condition number, and the method allows larger step-sizes than previous results.
• For asynchronous stochastic gradient descent, the analysis provides tighter convergence guarantees that depend on the average delay rather than the maximum delay, improving practicality and scalability.
• The ARock framework for asynchronous block-coordinate updates achieves faster convergence rates and a larger range of admissible step-sizes due to the new sequence-based analysis.
• For totally asynchronous iterations with unbounded delays growing linearly, the convergence rate is shown to be O(1/k^η) for some η > 0, establishing explicit decay under realistic delay models.
• The framework recovers and strengthens asymptotic convergence results from prior work while providing explicit, quantifiable bounds, demonstrating its generality and power.