[Paper Review] Reinforcement Learning for Mean Field Games, with Applications to Economics

Andrea Angiuli, Jean‐Pierre Fouque+1 more|arXiv (Cornell University)|Jun 25, 2021

Economic Policies and Impacts4 citations

TL;DR

This paper proposes a two-timescale reinforcement learning algorithm, U2-MF-QL-FH, for solving mean field games (MFG) and mean field control (MFC) problems with control-based interactions in finite-horizon settings. By decoupling the learning rates for the Q-function and the control distribution, the method unifies MFG and MFC learning in a model-free framework, achieving accurate approximations of both equilibrium and optimal control solutions through Q-learning and distribution updates.

ABSTRACT

Mean field games (MFG) and mean field control problems (MFC) are frameworks to study Nash equilibria or social optima in games with a continuum of agents. These problems can be used to approximate competitive or cooperative games with a large finite number of agents and have found a broad range of applications, in particular in economics. In recent years, the question of learning in MFG and MFC has garnered interest, both as a way to compute solutions and as a way to model how large populations of learners converge to an equilibrium. Of particular interest is the setting where the agents do not know the model, which leads to the development of reinforcement learning (RL) methods. After reviewing the literature on this topic, we present a two timescale approach with RL for MFG and MFC, which relies on a unified Q-learning algorithm. The main novelty of this method is to simultaneously update an action-value function and a distribution but with different rates, in a model-free fashion. Depending on the ratio of the two learning rates, the algorithm learns either the MFG or the MFC solution. To illustrate this method, we apply it to a mean field problem of accumulated consumption in finite horizon with HARA utility function, and to a trader's optimal liquidation problem.

Motivation & Objective

• To develop a model-free reinforcement learning method for solving mean field games and mean field control problems with interaction through the distribution of controls.
• To unify MFG and MFC solution learning under a single algorithmic framework using different learning rate ratios.
• To demonstrate the method’s effectiveness on two economic applications: HARA utility-based consumption and optimal liquidation.
• To provide a scalable, numerically robust approach for large-population decision problems where agents lack full model knowledge.
• To extend RL-based mean field solution methods beyond known dynamics, enabling learning from simulated or black-box environments.

Proposed method

• The algorithm uses a two-timescale update rule: faster updates for the action-value (Q-matrix) and slower updates for the empirical distribution of controls.
• It employs a Q-learning update for the Q-function based on observed rewards and state transitions, using ε-greedy exploration with ε = 0.1.
• The distribution of controls is updated using a moving average or empirical frequency estimate, with a slower learning rate than the Q-function.
• The relative ratio of learning rates (ω^Q, ω^θ) determines whether the algorithm converges to an MFG or MFC solution: faster Q-updating favors MFG, slower favors MFC.
• The method is applied to discretized continuous-state problems, with state and action spaces quantized using Δx = Δa = 1/4.
• Convergence is justified via Borkar’s stochastic approximation theory, extended to mean field settings.

Experimental results

Research questions

• RQ1 Can a single reinforcement learning algorithm approximate both mean field game and mean field control solutions by adjusting learning rate ratios?
• RQ2 How does the relative speed of Q-function versus distribution update affect convergence to MFG versus MFC solutions?
• RQ3 To what extent can a model-free RL approach learn optimal controls and mean field terms when the underlying dynamics and reward functions are unknown?
• RQ4 How does the algorithm perform on economically relevant problems such as HARA utility maximization and optimal liquidation?
• RQ5 What is the impact of discretization and exploration strategy on the accuracy of learned control and distribution functions?

Key findings

• The U2-MF-QL-FH algorithm successfully approximates the theoretical MFG solution when (ω^Q, ω^θ) = (0.55, 0.85), with close alignment between learned and theoretical controls at early times.
• When (ω^Q, ω^θ) = (0.65, 0.15), the algorithm converges to the MFC solution, demonstrating the method’s ability to switch between equilibrium and optimal control regimes.
• Approximation accuracy is highest at initial time points and degrades toward the final horizon, indicating higher complexity in late-stage learning.
• The learned mean of the control distribution converges to the theoretical mean in both MFG and MFC cases, with results averaged over 10 runs showing stable convergence trends.
• The method achieves stable learning across 10 runs for both the capital accumulation and liquidation problems, with consistent behavior across time points.
• The algorithm demonstrates robustness to model uncertainty, functioning as a numerical solver even when dynamics and rewards are known but complex.