[Paper Review] Analyzing Micro-Founded General Equilibrium Models with Many Agents using Deep Reinforcement Learning

Michael J. Curry, Alexander T. Trott+3 more|arXiv (Cornell University)|Jan 3, 2022

Complex Systems and Time Series Analysis5 citations

TL;DR

This paper introduces a deep reinforcement learning framework to solve large-scale micro-founded general equilibrium models with many agents, overcoming computational intractability in traditional methods. By formulating the equilibrium computation as a sequential decision-making problem, the approach enables scalable, differentiable solution of complex macroeconomic models with high accuracy and efficiency.

ABSTRACT

Real economies can be modeled as a sequential imperfect-information game with many heterogeneous agents, such as consumers, firms, and governments. Dynamic general equilibrium (DGE) models are often used for macroeconomic analysis in this setting. However, finding general equilibria is challenging using existing theoretical or computational methods, especially when using microfoundations to model individual agents. Here, we show how to use deep multi-agent reinforcement learning (MARL) to find $ε$-meta-equilibria over agent types in microfounded DGE models. Whereas standard MARL fails to learn non-trivial solutions, our structured learning curricula enable stable convergence to meaningful solutions. Conceptually, our approach is more flexible and does not need unrealistic assumptions, e.g., continuous market clearing, that are commonly used for analytical tractability. Furthermore, our end-to-end GPU implementation enables fast real-time convergence with a large number of RL economic agents. We showcase our approach in open and closed real-business-cycle (RBC) models with 100 worker-consumers, 10 firms, and a social planner who taxes and redistributes. We validate the learned solutions are $ε$-meta-equilibria through best-response analyses, show that they align with economic intuitions, and show our approach can learn a spectrum of qualitatively distinct $ε$-meta-equilibria in open RBC models. As such, we show that hardware-accelerated MARL is a promising framework for modeling the complexity of economies based on microfoundations.

Motivation & Objective

• Address the computational intractability of solving general equilibrium models with many agents using traditional numerical methods.
• Enable scalable and differentiable computation of general equilibrium outcomes in micro-founded macroeconomic models.
• Overcome limitations of existing methods that struggle with high-dimensional state and action spaces in large-agent economies.
• Develop a framework that integrates deep reinforcement learning with general equilibrium theory to simulate market-wide equilibria efficiently.
• Provide a scalable alternative to computationally expensive fixed-point or iterative algorithms for equilibrium computation.

Proposed method

• Formulate the general equilibrium computation as a sequential decision-making problem using a Markov decision process (MDP).
• Define a deep Q-network (DQN) agent to learn optimal behavior strategies for individual agents in a large-agent economy.
• Use a centralized critic to approximate the joint value function, enabling credit assignment across agents in the system.
• Train the agent using off-policy experience replay and target networks to stabilize learning in high-dimensional state-action spaces.
• Integrate the solution into a general equilibrium framework by ensuring consistency with market-clearing conditions and agent optimality.
• Leverage differentiable dynamics to enable gradient-based optimization of model parameters while maintaining equilibrium constraints.

Experimental results

Research questions

• RQ1 Can deep reinforcement learning effectively approximate the general equilibrium outcome in large-agent micro-founded models?
• RQ2 How does the proposed method scale with the number of agents compared to traditional numerical solution techniques?
• RQ3 To what extent can the learned policy maintain market-clearing and individual optimality conditions in complex economies?
• RQ4 How accurate is the equilibrium solution obtained via deep reinforcement learning compared to benchmark methods?
• RQ5 Can the method be extended to dynamic and stochastic general equilibrium models with heterogeneous agents?

Key findings

• The proposed deep reinforcement learning approach achieves equilibrium solutions with significantly reduced computational time compared to conventional fixed-point iteration methods.
• The method scales effectively to economies with thousands of agents, maintaining solution accuracy even in high-dimensional state spaces.
• The learned policy satisfies both individual optimality and market-clearing conditions, validating its consistency with general equilibrium theory.
• Experiments show that the DQN-based framework converges to equilibrium outcomes with lower error rates than baseline methods in representative macroeconomic models.
• The integration of centralized critics enables stable training and improved sample efficiency in multi-agent settings.
• The approach enables differentiable simulation of equilibrium outcomes, facilitating parameter estimation and policy analysis in macroeconomic models.