[Paper Review] APAC-Net: Alternating the Population and Agent Control via Two Neural Networks to Solve High-Dimensional Stochastic Mean Field Games

A. T. Lin, Samy Wu Fung+3 more|arXiv (Cornell University)|Jan 1, 2020

Model Reduction and Neural Networks19 references26 citations

TL;DR

APAC-Net introduces a novel neural network framework that alternates between optimizing population and agent controls in high-dimensional stochastic mean field games (MFGs) by casting the problem as a convex-concave saddle point via a primal-dual variational structure. By parameterizing value and density functions with two separate neural networks, the method reformulates MFG solution as a GAN-like training process, successfully solving MFGs up to 100 dimensions—previously intractable with existing methods.

ABSTRACT

We present APAC-Net, an alternating population and agent control neural network for solving stochastic mean field games (MFGs). Our algorithm is geared toward high-dimensional instances of MFGs that are beyond reach with existing solution methods. We achieve this in two steps. First, we take advantage of the underlying variational primal-dual structure that MFGs exhibit and phrase it as a convex-concave saddle point problem. Second, we parameterize the value and density functions by two neural networks, respectively. By phrasing the problem in this manner, solving the MFG can be interpreted as a special case of training a generative adversarial network (GAN). We show the potential of our method on up to 100-dimensional MFG problems.

Motivation & Objective

• To address the challenge of solving high-dimensional stochastic mean field games (MFGs), which remain computationally intractable with traditional methods.
• To exploit the variational primal-dual structure inherent in MFGs to reformulate the problem as a convex-concave saddle point optimization.
• To develop a scalable deep learning framework that parameterizes both the value function and the population density using two distinct neural networks.
• To enable end-to-end training of the system by drawing parallels to generative adversarial networks (GANs), facilitating stable optimization in high-dimensional spaces.
• To demonstrate the method’s effectiveness on MFG problems with up to 100 dimensions, pushing the boundary of existing solution capabilities.

Proposed method

• The method formulates the stochastic MFG as a convex-concave saddle point problem by leveraging its underlying variational primal-dual structure.
• It parameterizes the value function and the population density using two separate neural networks, enabling scalable representation in high-dimensional spaces.
• The training process alternates between optimizing the agent control (value network) and the population control (density network), mimicking the alternating training in GANs.
• The algorithm treats the solution of the MFG as a minimax optimization problem, where the value network minimizes and the density network maximizes a shared objective function.
• The framework ensures convergence by exploiting the convex-concave nature of the reformulated problem, enabling stable training even in high dimensions.
• The method is implemented end-to-end using deep learning backpropagation, allowing gradient-based optimization of both neural networks simultaneously.

Experimental results

Research questions

• RQ1 Can a deep learning framework effectively solve high-dimensional stochastic mean field games beyond the reach of classical numerical methods?
• RQ2 How can the primal-dual variational structure of MFGs be leveraged to design a stable and scalable optimization algorithm?
• RQ3 To what extent can the MFG solution process be interpreted as a generative adversarial network (GAN)-like training procedure?
• RQ4 What is the maximum dimensionality of MFG problems that can be solved using a neural network-based alternating optimization scheme?
• RQ5 How does the alternating optimization between population and agent control networks improve convergence and solution quality in high-dimensional settings?

Key findings

• APAC-Net successfully solves stochastic mean field games with up to 100 dimensions, demonstrating scalability beyond the limits of prior methods.
• The reformulation of MFGs as a convex-concave saddle point problem enables stable and efficient optimization using neural network parameterization.
• The method achieves effective learning by alternating between optimizing the value function and the population density, emulating the training dynamics of GANs.
• The use of two distinct neural networks for value and density functions allows for more expressive and accurate representations in high-dimensional state spaces.
• The framework shows promise in handling complex, high-dimensional stochastic dynamics where traditional solvers fail due to the curse of dimensionality.
• The approach provides a new paradigm for solving MFGs by unifying variational principles with deep learning, opening avenues for future research in large-scale stochastic control.