[Paper Review] Online Game with Time-Varying Coupled Inequality Constraints

Min Meng, Xiuxian Li+3 more|arXiv (Cornell University)|Jun 28, 2023

Advanced Bandit Algorithms ResearchDecision Sciences3 citations

TL;DR

This paper proposes a decentralized online learning algorithm for games with time-varying coupled inequality constraints, using mirror descent and primal-dual updates to achieve sublinear regret and constraint violation. It proves convergence to the variational generalized Nash equilibrium under strong monotonicity, with extensions to bandit feedback settings.

ABSTRACT

In this paper, online game is studied, where at each time, a group of players aim at selfishly minimizing their own time-varying cost function simultaneously subject to time-varying coupled constraints and local feasible set constraints. Only local cost functions and local constraints are available to individual players, who can share limited information with their neighbors through a fixed and connected graph. In addition, players have no prior knowledge of future cost functions and future local constraint functions. In this setting, a novel decentralized online learning algorithm is devised based on mirror descent and a primal-dual strategy. The proposed algorithm can achieve sublinearly bounded regrets and constraint violation by appropriately choosing decaying stepsizes. Furthermore, it is shown that the generated sequence of play by the designed algorithm can converge to the variational GNE of a strongly monotone game, to which the online game converges. Additionally, a payoff-based case, i.e., in a bandit feedback setting, is also considered and a new payoff-based learning policy is devised to generate sublinear regrets and constraint violation. Finally, the obtained theoretical results are corroborated by numerical simulations.

Motivation & Objective

• To address online games with time-varying cost functions and coupled constraints in decentralized settings where players lack full information about future costs or global constraints.
• To design a decentralized online learning algorithm that operates under partial information, relying only on local cost functions, local constraints, and neighbor communication via a fixed connected graph.
• To establish theoretical guarantees on sublinear regret and constraint violation in the presence of time-varying constraints and limited feedback.
• To extend the framework to a bandit feedback setting where only payoff (function value) information is available, not gradients.
• To prove convergence of the generated play sequence to the variational generalized Nash equilibrium under strong monotonicity assumptions.

Proposed method

• A novel decentralized online learning algorithm is designed using mirror descent and a primal-dual strategy to handle time-varying coupled inequality constraints and local feasible sets.
• The algorithm employs decaying stepsizes to balance regret and constraint violation, ensuring sublinear growth over time.
• The method incorporates local gradient estimates and dual variable updates to track time-varying constraints, with convergence analyzed via Lyapunov-like functions and recursive inequalities.
• For the bandit feedback case, a new payoff-based learning policy is introduced that infers gradient information from single-point observations, enabling regret and constraint violation bounds without access to gradients.
• Key components include the use of decaying sequences for stepsizes (αₜ, βₜ, γₜ, δₜ), dual variable tracking, and error decomposition using conditional expectations and Lipschitz continuity assumptions.
• Theoretical analysis leverages inequalities involving conditional expectations and bounds on gradient and constraint function estimation errors, leading to regret and violation bounds in terms of tuning parameters.

Experimental results

Research questions

• RQ1 Can a decentralized online learning algorithm achieve sublinear regret and constraint violation in a time-varying online game with coupled constraints and limited information?
• RQ2 How can players converge to the variational generalized Nash equilibrium (v-GNE) in a decentralized online game with time-varying constraints and no prior knowledge of future costs?
• RQ3 What is the performance guarantee of the algorithm in the bandit feedback setting, where only function values are observed and gradients are not available?
• RQ4 How do decaying stepsizes affect the trade-off between regret and constraint violation in the proposed algorithm?
• RQ5 Under what conditions does the sequence of play converge to the v-GNE, and what role does strong monotonicity play in this convergence?

Key findings

• The proposed algorithm achieves sublinear regret bounded by 𝒪(∑(δᵢ,ₜ + γₜ + αₜ/δᵢ,ₜ + αₜ/βₜ² + δₜ/βₜ) + 1/αₜ) under full information feedback.
• Constraint violation is bounded by 𝒪(∑(δₜ/βₜ) + (𝔼[R_g(T)])²/B₃(T)), with R_g(T) denoting the total constraint violation.
• When stepsizes are chosen as αₜ = 𝒪(1/√t), βₜ = 𝒪(1/t), and δₜ = 𝒪(1/t), the regret and constraint violation both grow sublinearly with time T.
• The sequence of play generated by the algorithm converges to the variational generalized Nash equilibrium (v-GNE) of the underlying game under strong monotonicity.
• In the bandit feedback setting, the payoff-based learning policy achieves sublinear regret and constraint violation despite only observing function values, not gradients.
• Numerical simulations corroborate the theoretical findings, demonstrating the effectiveness of the algorithm in achieving low regret and constraint violation over time.