[Paper Review] Near-Optimal No-Regret Learning Dynamics for General Convex Games

Gabriele Farina, Ioannis Anagnostides+4 more|arXiv (Cornell University)|Jun 17, 2022

Advanced Bandit Algorithms Research4 citations

TL;DR

This paper introduces LRL-OFTRL, a novel no-regret learning algorithm for general convex games with arbitrary convex and compact strategy sets. By leveraging an optimistic follow-the-regularized-leader framework over a lifted space with a self-concordant regularizer (not a barrier), it achieves $O(\log T)$ per-player regret, significantly improving upon traditional $O(\sqrt{T})$ bounds while remaining efficiently implementable via a proximal oracle with $O(\log\log T)$ per-iteration complexity.

ABSTRACT

A recent line of work has established uncoupled learning dynamics such that, when employed by all players in a game, each player's \emph{regret} after $T$ repetitions grows polylogarithmically in $T$, an exponential improvement over the traditional guarantees within the no-regret framework. However, so far these results have only been limited to certain classes of games with structured strategy spaces -- such as normal-form and extensive-form games. The question as to whether $O( ext{polylog} T)$ regret bounds can be obtained for general convex and compact strategy sets -- which occur in many fundamental models in economics and multiagent systems -- while retaining efficient strategy updates is an important question. In this paper, we answer this in the positive by establishing the first uncoupled learning algorithm with $O(\log T)$ per-player regret in general \emph{convex games}, that is, games with concave utility functions supported on arbitrary convex and compact strategy sets. Our learning dynamics are based on an instantiation of optimistic follow-the-regularized-leader over an appropriately \emph{lifted} space using a \emph{self-concordant regularizer} that is, peculiarly, not a barrier for the feasible region. Further, our learning dynamics are efficiently implementable given access to a proximal oracle for the convex strategy set, leading to $O(\log\log T)$ per-iteration complexity; we also give extensions when access to only a \emph{linear} optimization oracle is assumed. Finally, we adapt our dynamics to guarantee $O(\sqrt{T})$ regret in the adversarial regime. Even in those special cases where prior results apply, our algorithm improves over the state-of-the-art regret bounds either in terms of the dependence on the number of iterations or on the dimension of the strategy sets.

Motivation & Objective

• To close the gap in no-regret learning for general convex games, where prior results were limited to structured games like normal-form or extensive-form games.
• To develop an efficiently implementable, strongly uncoupled learning dynamics that achieves near-optimal $O(\log T)$ regret in general convex games with arbitrary convex and compact strategy sets.
• To extend the state-of-the-art $O(\operatorname{polylog}T)$ regret guarantees to broader classes of games, including routing, resource allocation, and market competition models.
• To ensure practical efficiency by requiring only a proximal oracle (weaker than quadratic or linear oracles), with $O(\log\log T)$ per-iteration complexity.
• To maintain robustness by guaranteeing $O(\sqrt{T})$ regret in the adversarial regime, even when utilities are non-stationary.

Proposed method

• Propose a novel learning dynamics, LRL-OFTRL, based on optimistic follow-the-regularized-leader (OFTRL) over a lifted space of strategy variables and dual variables.
• Use a self-concordant regularizer that is not a barrier function for the feasible region, enabling efficient computation and strong regret guarantees.
• Lift the primal strategy variables into a higher-dimensional space where the regularizer allows for closed-form updates and KKT system simplification.
• Derive the optimal dual variables via a KKT system that reduces to solving for the subgradient of a value function $V_Q(t; \bm{g}, \bm{w})$, which is shown to be a piecewise-linear (SMPL) function.
• Construct a proximal oracle by computing the optimal $t \in [0,1]$ that minimizes the regularized objective, using the derivative $\lambda_Q(t; \bm{g}, \bm{w})$ and leveraging SMPL structure for efficient computation.
• Extend the method to work under a weaker linear optimization oracle, albeit with higher per-iteration complexity, ensuring broader applicability.

Experimental results

Research questions

• RQ1 Can $O(\log T)$ regret be achieved in general convex games with arbitrary convex and compact strategy sets using an efficiently implementable, uncoupled learning algorithm?
• RQ2 Can such a regret guarantee be maintained when utilities are adversarial, ensuring robustness in non-stationary environments?
• RQ3 Can the algorithm be efficiently implemented using only a proximal oracle, avoiding the need for stronger quadratic oracles?
• RQ4 Does the proposed method outperform prior $O(\operatorname{polylog}T)$ regret algorithms in terms of regret dependence on $T$ or dimensionality?
• RQ5 Can the lifting and self-concordant regularizer framework be used to derive a closed-form or efficiently computable update rule for the strategy space?

Key findings

• The proposed LRL-OFTRL algorithm achieves $O(\log T)$ per-player regret in general convex games, representing a significant improvement over the classical $O(\sqrt{T})$ bound.
• The regret bound is optimal up to logarithmic factors, matching the best-known guarantees in structured games like normal-form games.
• The algorithm is efficiently implementable with only $O(\log\log T)$ operations per iteration when a proximal oracle is available.
• The method remains robust, guaranteeing $O(\sqrt{T})$ regret in the adversarial regime, even when utilities are non-stationary.
• The proximal oracle construction relies on the SMPL (piecewise-linear) structure of the dual derivative $\lambda_Q(t; \bm{g}, \bm{w})$, enabling exact and efficient minimization over $t \in [0,1]$.
• The approach extends to settings with only a linear optimization oracle, albeit with a higher per-iteration complexity of $O(\log T)$, broadening its applicability.