[Paper Review] A Distributed Online Convex Optimization Algorithm with Improved Dynamic Regret

Yan Zhang, Robert Ravier+2 more|arXiv (Cornell University)|Nov 12, 2019

Advanced Bandit Algorithms Research18 references4 citations

TL;DR

This paper proposes a distributed online convex optimization algorithm using gradient tracking to achieve dynamic regret without explicit dependence on the number of time steps, under strong convexity. By correcting gradient steps through local communication, the method ensures sublinear regret that scales only with path-length regularity, outperforming prior methods in convergence and stepsize robustness.

ABSTRACT

In this paper, we consider the problem of distributed online convex optimization, where a network of local agents aim to jointly optimize a convex function over a period of multiple time steps. The agents do not have any information about the future. Existing algorithms have established dynamic regret bounds that have explicit dependence on the number of time steps. In this work, we show that we can remove this dependence assuming that the local objective functions are strongly convex. More precisely, we propose a gradient tracking algorithm where agents jointly communicate and descend based on corrected gradient steps. We verify our theoretical results through numerical experiments.

Motivation & Objective

• To address the limitation of existing distributed online convex optimization algorithms that exhibit dynamic regret growing with the number of time steps.
• To remove explicit time dependence in dynamic regret bounds by leveraging strong convexity and gradient tracking.
• To design a distributed algorithm that enables faster convergence and higher stepsize tolerance compared to prior methods.
• To validate the theoretical improvements through numerical experiments in a distributed tracking scenario.

Proposed method

• The algorithm employs a gradient tracking mechanism where agents maintain local estimates of the global gradient and correct their descent steps using information from neighbors.
• Each agent updates its local variable using a fixed step size and a weighted average of its own and neighbors' gradients, ensuring consensus on the gradient estimate.
• The method incorporates both optimizer path-length and gradient path-length regularities to bound dynamic regret.
• A key component is the use of a doubly stochastic weight matrix to ensure convergence across the network.
• The algorithm is derived as an online extension of decentralized gradient descent with gradient correction, inspired by recent offline advances.
• Theoretical analysis establishes that dynamic regret scales as O(√P_T) without explicit T dependence under strong convexity.

Experimental results

Research questions

• RQ1 Can dynamic regret in distributed online convex optimization be bounded without explicit dependence on the number of time steps?
• RQ2 How does strong convexity enable improved regret bounds in distributed online settings?
• RQ3 Can gradient tracking techniques be adapted to online settings to improve convergence and stepsize robustness?
• RQ4 What is the impact of gradient path-length regularity on the performance of distributed online algorithms?
• RQ5 How does the proposed algorithm compare to existing methods in terms of regret and tracking accuracy in dynamic environments?

Key findings

• The proposed algorithm achieves dynamic regret bounded by O(√P_T), where P_T is the path-length of optimal solutions, without explicit dependence on the number of time steps T.
• The regret bound scales only with the path-length regularity, demonstrating that strong convexity eliminates the need for time-dependent terms in the bound.
• Numerical experiments show that the algorithm achieves lower regret and better tracking performance than the baseline distributed online gradient descent without gradient tracking.
• The algorithm allows for larger step sizes than existing methods, with the ability to maintain stability and convergence even at stepsizes of 1/(2L_g), where the baseline diverges.
• In distributed tracking simulations, agents successfully track multiple moving targets with minimal error, and the regret grows at the same rate as the path-length regularity.
• The gradient path-length regularity V_T was found to be of the same order as P_T in the tracking example, validating the theoretical assumptions.