[Paper Review] Reinforcement Learning for Combining Search Methods in the Calibration of Economic ABMs

Aldo Glielmo, Marco Favorito+2 more|arXiv (Cornell University)|Feb 23, 2023

Complex Systems and Time Series AnalysisEconomics, Econometrics and Finance56 references3 citations

TL;DR

This paper proposes a reinforcement learning (RL) framework that dynamically combines multiple search methods for calibrating economic agent-based models (ABMs), outperforming individual or static combinations. By treating method selection as a multi-armed bandit problem, the RL agent adapts in real time, exploiting high-performing strategies and exploring alternatives at plateaus, achieving superior calibration efficiency on benchmark ABMs without prior tuning or assumptions.

ABSTRACT

Calibrating agent-based models (ABMs) in economics and finance typically involves a derivative-free search in a very large parameter space. In this work, we benchmark a number of search methods in the calibration of a well-known macroeconomic ABM on real data, and further assess the performance of "mixed strategies" made by combining different methods. We find that methods based on random-forest surrogates are particularly efficient, and that combining search methods generally increases performance since the biases of any single method are mitigated. Moving from these observations, we propose a reinforcement learning (RL) scheme to automatically select and combine search methods on-the-fly during a calibration run. The RL agent keeps exploiting a specific method only as long as this keeps performing well, but explores new strategies when the specific method reaches a performance plateau. The resulting RL search scheme outperforms any other method or method combination tested, and does not rely on any prior information or trial and error procedure.

Motivation & Objective

• To evaluate and compare the performance of diverse search methods in calibrating a well-known macroeconomic ABM using real data.
• To investigate whether combining multiple search strategies can mitigate individual method biases and improve calibration efficiency.
• To develop an automated, adaptive method selection framework that dynamically chooses the best search strategy during calibration.
• To demonstrate that reinforcement learning can outperform both individual search methods and fixed combinations without prior hyperparameter tuning.

Proposed method

• The RL agent frames method selection as a multi-armed bandit problem, where each search method is an 'arm' with a stochastic reward signal.
• The agent uses an epsilon-greedy strategy to balance exploration and exploitation, switching methods when performance plateaus.
• Reward is defined as the improvement in loss (e.g., method of moments or Euclidean distance) per model evaluation.
• The RL agent operates on-the-fly during calibration, updating its policy based on real-time feedback from ABM simulations.
• The framework is evaluated on two ABMs: the BH4 model (Brock and Hommes, 1998) and the SIR model, using both method-of-moments and Euclidean distance loss functions.
• The approach does not require prior knowledge of the optimal method or trial-and-error tuning, making it robust and generalizable.

Experimental results

Research questions

• RQ1 Which individual search method performs best in calibrating a standard macroeconomic ABM on real data?
• RQ2 Can combining multiple search methods improve calibration performance beyond any single method?
• RQ3 How can an adaptive strategy be designed to automatically select the best search method during calibration?
• RQ4 Does a reinforcement learning-based method selection scheme outperform static combinations and individual methods in terms of convergence speed and accuracy?

Key findings

• Random-forest-based surrogate methods outperformed other individual search strategies in terms of convergence speed and accuracy on the benchmark ABMs.
• Combining search methods consistently improved performance by mitigating the biases inherent in any single method.
• The proposed reinforcement learning scheme achieved the best overall performance, converging faster and reaching lower loss values than any tested method or combination.
• The RL agent successfully identified and exploited high-performing strategies while exploring alternatives at plateaus, without requiring prior hyperparameter tuning.
• The method demonstrated robustness across different loss functions (method of moments and Euclidean distance) and ABM types (BH4 and SIR models).
• The framework is generalizable and does not rely on domain-specific assumptions or pre-tuning, making it suitable for broad application in ABM calibration.