[Paper Review] Using Reinforcement Learning to Validate Empirical Game-Theoretic Analysis: A Continuous Double Auction Study

Mason Wright|arXiv (Cornell University)|Apr 22, 2016

Reinforcement Learning in Robotics22 references3 citations

TL;DR

This paper proposes using reinforcement learning (RL) to validate the strategic stability of equilibrium strategy profiles identified via Empirical Game-Theoretic Analysis (EGTA) in a continuous double auction market. By training diverse RL algorithms—including Q-learning, Sarsa, and POMCP—on EGTA-derived equilibria, the study demonstrates that no significant payoff improvements are found, providing strong empirical evidence that the EGTA profiles have negligible regret, even when tested against broader strategy spaces.

ABSTRACT

Empirical game-theoretic analysis (EGTA) has recently been applied successfully to analyze the behavior of large numbers of competing traders in a continuous double auction market. Multiagent simulation methods like EGTA are useful for studying complex strategic environments like a stock market, where it is not feasible to solve analytically for the rational behavior of each agent. A weakness of simulation-based methods in strategic settings, however, is that it is typically impossible to prove that the strategy profile assigned to the simulated agents is stable, as in a Nash equilibrium. I propose using reinforcement learning to analyze the regret of supposed Nash-equilibrium strategy profiles found by EGTA. I have developed a new library of reinforcement learning tools, which I have integrated into an extended version of the market simulator from our prior work. I provide evidence for the effectiveness of our library methods, both on a suite of benchmark problems from the literature, and on non-equilibrium strategy profiles in our market environment. Finally, I use our new reinforcement learning tools to provide evidence that the equilibria found by EGTA in our recent continuous double auction study are likely to have only negligible regret, even with respect to an extended strategy space.

Motivation & Objective

• To address the limitation of EGTA in verifying whether strategy profiles are truly stable Nash equilibria due to restricted strategy space sampling.
• To develop and validate a suite of reinforcement learning (RL) algorithms capable of detecting beneficial deviations from EGTA-derived strategy profiles.
• To provide empirical evidence that EGTA-identified equilibria in a continuous double auction environment are robust, even when tested against a broader policy space.
• To evaluate the effectiveness of RL methods in detecting non-equilibrium behavior and confirming the stability of equilibrium profiles through multiple algorithmic approaches.

Proposed method

• Implemented a library of RL tools, including Q-learning, Sarsa(λ), and POMCP, with enhancements like tile coding, experience replay, and eligibility traces to improve learning stability and speed.
• Integrated the RL library into an extended version of a prior market simulator to enable testing of EGTA strategy profiles in a realistic continuous double auction environment.
• Conducted experiments comparing RL agents' performance against both non-equilibrium strategy profiles (to validate RL efficacy) and EGTA equilibrium profiles (to test stability).
• Allowed RL agents to choose between buying or selling actions to test whether they could outperform fixed EGTA policies, simulating strategic flexibility.
• Used multiple RL algorithms with varying mechanisms (tabular vs. Monte Carlo tree search) to increase confidence in results and reduce bias from any single method.
• Evaluated RL implementations on benchmark problems (e.g., RockSample, Sutton’s gridworld) to ensure correctness before applying them to the market simulation.

Experimental results

Research questions

• RQ1 Can reinforcement learning detect profitable deviations from non-equilibrium strategy profiles in a continuous double auction setting?
• RQ2 Can reinforcement learning fail to find better strategies when playing against EGTA-identified equilibrium profiles, suggesting low regret?
• RQ3 How do different RL algorithms (e.g., Q-learning vs. POMCP) compare in performance and computational efficiency when testing equilibrium stability?
• RQ4 Does allowing RL agents to choose between buying or selling actions lead to higher payoffs than fixed EGTA policies, indicating potential instability?
• RQ5 To what extent can empirical RL results support the claim that EGTA profiles are strategically stable, even when the full strategy space is not explored?

Key findings

• All RL methods—Q-learning, Sarsa(λ), and POMCP—reached a performance plateau at or near the payoff level of the EGTA equilibrium profiles in all tested environments, indicating no significant improvement could be found.
• In the FlipKnown setting, POMCP did not outperform the EGTA equilibrium in any environment, with 95% confidence intervals overlapping the equilibrium payoff, and in one case (B-1k-eq), POMCP performed significantly worse.
• When allowed to choose between buying or selling, RL agents consistently achieved higher mean payoffs than the fixed EGTA policy, suggesting the EGTA profile is not easily exploitable.
• The RL methods successfully improved upon non-equilibrium profiles, confirming their ability to detect and exploit strategic weaknesses.
• POMCP was significantly less efficient than classical RL methods due to high computational cost, resulting in wide confidence intervals and limited sample sizes, especially at higher playouts.
• The combination of multiple RL algorithms, validated on benchmark problems and shown to detect non-equilibrium behavior, provides strong empirical support for the strategic stability of the EGTA profiles.