[Paper Review] Dynamic Regret Analysis of Safe Distributed Online Optimization for Convex and Non-convex Problems

T. T. Chang, Sapana Chaudhary+2 more|arXiv (Cornell University)|Feb 23, 2023

Stochastic Gradient Optimization Techniques4 citations

TL;DR

This paper proposes D-Safe-OGD, a distributed online optimization algorithm that ensures safety under unknown linear constraints while minimizing dynamic regret. It combines collaborative constraint estimation with online gradient descent, achieving a dynamic regret bound of $O(T^{2/3} ext{polylog}(T) + T^{1/3}C_T^*)$ for convex problems and $O(T^{2/3} ext{polylog}(T) + T^{2/3}C_T^*)$ for certain non-convex problems, establishing the first such bound in the non-convex safe distributed setting.

ABSTRACT

This paper addresses safe distributed online optimization over an unknown set of linear safety constraints. A network of agents aims at jointly minimizing a global, time-varying function, which is only partially observable to each individual agent. Therefore, agents must engage in local communications to generate a safe sequence of actions competitive with the best minimizer sequence in hindsight, and the gap between the two sequences is quantified via dynamic regret. We propose distributed safe online gradient descent (D-Safe-OGD) with an exploration phase, where all agents estimate the constraint parameters collaboratively to build estimated feasible sets, ensuring the action selection safety during the optimization phase. We prove that for convex functions, D-Safe-OGD achieves a dynamic regret bound of $O(T^{2/3} \sqrt{\log T} + T^{1/3}C_T^*)$, where $C_T^*$ denotes the path-length of the best minimizer sequence. We further prove a dynamic regret bound of $O(T^{2/3} \sqrt{\log T} + T^{2/3}C_T^*)$ for certain non-convex problems, which establishes the first dynamic regret bound for a safe distributed algorithm in the non-convex setting.

Motivation & Objective

• To address distributed online optimization under unknown linear safety constraints, where agents must jointly minimize a time-varying global objective.
• To ensure action safety during optimization despite partial observability of the global function and unknown constraints.
• To derive dynamic regret bounds that scale with the path-length of the optimal minimizer sequence $C_T^*$, reflecting adaptability to time-varying environments.
• To extend dynamic regret analysis to non-convex problems under safety constraints, a gap previously unaddressed in distributed online learning.

Proposed method

• Proposes D-Safe-OGD, a distributed online gradient descent algorithm with a dedicated exploration phase for collaborative estimation of unknown constraint parameters.
• Uses a network of agents to jointly estimate feasible sets via local communication, ensuring safety during optimization.
• Employs Bregman divergence and online mirror descent techniques to analyze regret, incorporating dual variables and projected updates.
• Introduces a two-phase strategy: an initial exploration phase to estimate constraints, followed by optimization with safety guarantees.
• Applies reparameterization and geometric assumptions to extend convex regret analysis to non-convex settings.
• Derives regret bounds using path-length $C_T^*$ as a measure of environmental variation, linking performance to temporal stability.

Experimental results

Research questions

• RQ1 Can a distributed online optimization algorithm maintain safety under unknown linear constraints while achieving sublinear dynamic regret?
• RQ2 What is the dynamic regret bound for convex distributed online optimization with unknown safety constraints?
• RQ3 Can the dynamic regret analysis be extended to non-convex problems under safety constraints in a distributed setting?
• RQ4 How does the path-length $C_T^*$ of the optimal sequence affect the regret bound in time-varying environments?
• RQ5 What is the trade-off between exploration for constraint estimation and regret minimization in distributed online learning?

Key findings

• For convex problems, D-Safe-OGD achieves a dynamic regret bound of $O(T^{2/3} ext{polylog}(T) + T^{1/3}C_T^*)$, where $C_T^*$ is the path-length of the optimal sequence.
• For certain non-convex problems, the algorithm achieves a dynamic regret bound of $O(T^{2/3} ext{polylog}(T) + T^{2/3}C_T^*)$, the first such result in the safe distributed non-convex setting.
• The regret bound scales sublinearly with time $T$, indicating convergence to the optimal sequence over time despite environmental changes.
• The exploration phase enables accurate estimation of unknown constraint parameters, ensuring safety without prior knowledge.
• The analysis leverages Bregman divergence and online mirror descent techniques to handle non-smooth and non-convex objectives.
• The derived bounds are tight in terms of $T$ and $C_T^*$, matching known lower bounds in the convex case and extending them to non-convex and safe settings.