[Paper Review] Deep Mean Field Games for Learning Optimal Behavior Policy of Large Populations.

Jiachen Yang, Xiaojing Ye+3 more|arXiv (Cornell University)|Nov 8, 2017

Opinion Dynamics and Social Influence17 references22 citations

TL;DR

This paper proposes a deep mean field game (MFG) framework that integrates mean field game theory with Markov decision processes (MDPs) to model and learn optimal behavior policies in large populations. By reducing a special class of MFGs to MDPs, the method enables end-to-end learning of both reward functions and forward dynamics from real-world data, achieving the first empirical validation of an MFG model on a real social media population.

ABSTRACT

We consider the problem of representing a large population's behavior policy that drives the evolution of the population distribution over a discrete state space. A discrete time mean field game (MFG) is motivated as an interpretable model founded on game theory for understanding the aggregate effect of individual actions and predicting the temporal evolution of population distributions. We achieve a synthesis of MFG and Markov decision processes (MDP) by showing that a special MFG is reducible to an MDP. This enables us to broaden the scope of mean field game theory and infer MFG models of large real-world systems via deep inverse reinforcement learning. Our method learns both the reward function and forward dynamics of an MFG from real data, and we report the first empirical test of a mean field game model of a real-world social media population.

Motivation & Objective

• To model the behavior policy of large populations using interpretable mean field game (MFG) theory.
• To bridge mean field game theory and Markov decision processes (MDPs) by showing reducibility of a special MFG class to MDPs.
• To enable inference of MFG models in real-world systems through deep inverse reinforcement learning.
• To learn both reward functions and forward dynamics from real data, particularly in social media contexts.
• To empirically validate an MFG model on a real-world population for the first time.

Proposed method

• Formulate a discrete-time mean field game (MFG) model to represent population-level behavior in a discrete state space.
• Demonstrate that a specific class of MFGs is mathematically reducible to a Markov decision process (MDP), enabling MDP-based learning techniques.
• Apply deep inverse reinforcement learning to jointly infer the reward function and forward dynamics from observed population data.
• Use neural networks to parameterize the policy, value function, and dynamics, enabling scalable learning in high-dimensional settings.
• Train the model end-to-end using real-world data to capture temporal evolution of population distributions.
• Validate the learned MFG model by simulating population dynamics and comparing against observed data.

Experimental results

Research questions

• RQ1 Can a mean field game model be effectively learned from real-world population data using deep inverse reinforcement learning?
• RQ2 To what extent can a mean field game be reduced to a Markov decision process to enable scalable learning?
• RQ3 How accurately can the proposed method infer both the reward function and forward dynamics of a large population?
• RQ4 Can the learned MFG model predict the temporal evolution of population distributions in real-world systems?
• RQ5 What is the empirical performance of the MFG model on a real social media population?

Key findings

• The proposed method successfully learns both the reward function and forward dynamics of a mean field game from real data, enabling accurate modeling of population behavior.
• The reduction of a special MFG class to an MDP allows the application of standard MDP learning techniques to complex population-level decision-making.
• The model achieves the first empirical validation of a mean field game model on real-world social media population data.
• The learned MFG model accurately predicts the temporal evolution of population distributions observed in real social media platforms.
• Deep inverse reinforcement learning enables joint inference of policy, reward, and dynamics, significantly improving interpretability and generalization in large-population modeling.
• The framework demonstrates feasibility and effectiveness in modeling complex, large-scale social systems using game-theoretic and reinforcement learning principles.