[Paper Review] First-Order Algorithms for Constrained Nonlinear Dynamic Games

Bolei Di, Andrew Lamperski|arXiv (Cornell University)|Jan 7, 2020

Optimization and Variational Analysis21 references4 citations

TL;DR

This paper presents first-order algorithms—projected gradient and Douglas-Rachford splitting—for computing open-loop Nash equilibria (OLNE) in constrained nonlinear dynamic games with linear convergence rates. It extends stagewise Newton methods to derive local feedback policies, showing they yield approximate feedback Nash equilibria under linear dynamics and polyhedral constraints, enabling efficient solution of multi-agent control problems with coupled constraints.

ABSTRACT

This paper presents algorithms for non-zero sum nonlinear constrained dynamic games with full information. Such problems emerge when multiple players with action constraints and differing objectives interact with the same dynamic system. They model a wide range of applications including economics, defense, and energy systems. We show how to exploit the temporal structure in projected gradient and Douglas-Rachford (DR) splitting methods. The resulting algorithms converge locally to open-loop Nash equilibria (OLNE) at linear rates. Furthermore, we extend stagewise Newton method to find a local feedback policy around an OLNE. In the of linear dynamics and polyhedral constraints, we show that this local feedback controller is an approximated feedback Nash equilibrium (FNE). Numerical examples are provided.

Motivation & Objective

• To develop scalable numerical algorithms for solving constrained non-zero-sum dynamic games with full information.
• To exploit temporal structure in first-order methods to achieve O(T) iteration complexity per step, where T is the horizon length.
• To extend open-loop Nash equilibrium computation to local feedback strategies using stagewise Newton methods.
• To establish conditions under which the resulting feedback policy approximates a feedback Nash equilibrium (FNE), particularly for linear dynamics and polyhedral constraints.
• To demonstrate convergence and performance through numerical examples, including a noisy biomass management game and a 3-player LQ rendezvous problem.

Proposed method

• Adapts projected gradient methods to constrained dynamic games by alternating gradient descent steps with projections onto joint action constraints, leveraging temporal structure for O(T) complexity.
• Applies Douglas-Rachford splitting to decompose the problem into an unconstrained game subproblem and a constraint projection subproblem, both solvable in O(T) time via dynamic programming.
• Uses stagewise Newton method to compute local feedback policies around computed OLNE trajectories, exploiting second-order structure for local optimality.
• For linear dynamics and polyhedral constraints, proves that the resulting feedback policy is an O(ε²)-approximate feedback Nash equilibrium (FNE).
• Employs a splitting scheme in Douglas-Rachford where one subproblem solves the unconstrained game via stagewise Newton and the other projects actions onto bounded sets.
• Implements both algorithms with line search and adaptive step size rules to ensure stable convergence in numerical experiments.

Experimental results

Research questions

• RQ1 Can first-order methods like projected gradient and Douglas-Rachford splitting be efficiently adapted to constrained nonlinear dynamic games with temporal structure?
• RQ2 What is the convergence rate of these algorithms when applied to constrained dynamic games, and can they maintain linear complexity in the horizon length?
• RQ3 Can the open-loop Nash equilibrium (OLNE) solution be used as a basis for computing a local feedback policy that approximates a feedback Nash equilibrium (FNE)?
• RQ4 Under what conditions—specifically, linear dynamics and polyhedral constraints—does the computed feedback policy yield a valid approximate FNE?
• RQ5 How do these algorithms perform in practice on benchmark problems such as a noisy biomass game and a multi-agent rendezvous task?

Key findings

• The projected gradient and Douglas-Rachford algorithms converge locally to open-loop Nash equilibria (OLNE) at linear rates, with O(T) iteration complexity per step.
• For the linear-quadratic rendezvous problem with convex constraints, the Douglas-Rachford method achieved stable convergence over 10,000 iterations, with the rendezvous point evolving toward the final OLNE.
• The norm of actions remained bounded by u_max = 2 throughout iterations, confirming constraint satisfaction.
• Cumulative costs for all three players decreased over iterations, indicating convergence to a lower-cost equilibrium.
• The local feedback policy derived via stagewise Newton method provided a smoother biomass trajectory in the noisy system, reducing deviation from the OLNE path.
• In the linear dynamics and polyhedral constraint case, the feedback policy was proven to be an O(ε²)-approximate feedback Nash equilibrium, validating its practical use.