[Paper Review] Reinforcement Learning in High-frequency Market Making

Yuheng Zheng, Zihan Ding|arXiv (Cornell University)|Jul 14, 2024

Innovation Diffusion and ForecastingDecision Sciences3 citations

TL;DR

This paper provides the first comprehensive theoretical analysis of reinforcement learning (RL) in high-frequency market making, establishing convergence of discrete-time RL algorithms to continuous-time models as sampling frequency increases. It reveals a tradeoff between learning error and computational complexity with respect to time increment Δ, and proves convergence of Nash equilibrium in a two-player general-sum game setting.

ABSTRACT

This paper establishes a new and comprehensive theoretical analysis for the application of reinforcement learning (RL) in high-frequency market making. We bridge the modern RL theory and the continuous-time statistical models in high-frequency financial economics. Different with most existing literature on methodological research about developing various RL methods for market making problem, our work is a pilot to provide the theoretical analysis. We target the effects of sampling frequency, and find an interesting tradeoff between error and complexity of RL algorithm when tweaking the values of the time increment $Δ$ $-$ as $Δ$ becomes smaller, the error will be smaller but the complexity will be larger. We also study the two-player case under the general-sum game framework and establish the convergence of Nash equilibrium to the continuous-time game equilibrium as $Δ ightarrow0$. The Nash Q-learning algorithm, which is an online multi-agent RL method, is applied to solve the equilibrium. Our theories are not only useful for practitioners to choose the sampling frequency, but also very general and applicable to other high-frequency financial decision making problems, e.g., optimal executions, as long as the time-discretization of a continuous-time markov decision process is adopted. Monte Carlo simulation evidence support all of our theories.

Motivation & Objective

• To provide a rigorous theoretical foundation for applying reinforcement learning (RL) to high-frequency market making, moving beyond heuristic methodological studies.
• To analyze the impact of sampling frequency (time increment Δ) on the accuracy and computational complexity of RL algorithms in continuous-time financial models.
• To extend the analysis to competitive market making scenarios involving multiple market makers using game-theoretic frameworks.
• To establish theoretical convergence of discrete-time RL solutions (e.g., Nash Q-learning) to their continuous-time counterparts as Δ → 0.
• To offer practical guidance for practitioners on selecting optimal sampling frequency based on tradeoffs between error and computational cost.

Proposed method

• Time-discretization of continuous-time Markov decision processes (MDPs) using a time increment Δ to enable application of discrete-time RL algorithms.
• Application of Q-learning for single-agent RL to solve the optimal market-making policy in the discretized MDP, with theoretical bounds on learning error.
• Extension to a two-player general-sum stochastic game model to capture competition among market makers, with Nash equilibrium as the solution concept.
• Development and theoretical analysis of the Nash Q-learning algorithm for multi-agent RL in the discretized game setting.
• Use of Monte Carlo simulations to validate theoretical convergence results and demonstrate empirical performance of the RL algorithms.
• Theoretical derivation of convergence rates for value functions and policies as Δ → 0, using controlled Markov chain theory and stochastic approximation.

Experimental results

Research questions

• RQ1 How does the choice of sampling frequency Δ affect the learning error and computational complexity in RL-based high-frequency market making?
• RQ2 What is the theoretical relationship between the discretized RL solution and the true continuous-time optimal policy as Δ → 0?
• RQ3 Does the Nash equilibrium of the discretized two-player game converge to the continuous-time game equilibrium as Δ → 0?
• RQ4 Can the Nash Q-learning algorithm effectively learn the equilibrium strategies in a multi-agent high-frequency market making environment?
• RQ5 How do the theoretical bounds on sample complexity and error scale with Δ in both single-agent and multi-agent settings?

Key findings

• A tradeoff exists between learning error and algorithmic complexity: as Δ decreases (higher sampling frequency), learning error decreases but computational cost increases.
• The value function error in the single-agent case converges to zero as Δ → 0, with empirical error dropping to (0.09, 0.05) in simulations.
• The policy error in the Nash Q-learning algorithm reaches zero in simulations, indicating exact recovery of the true equilibrium policy.
• The convergence of the Nash equilibrium strategy and value function from the discretized model to the continuous-time model is empirically validated, with |VΔ − V0| < 0.057.
• Theoretical analysis confirms that the Nash equilibrium of the discrete-time game converges to the continuous-time game equilibrium as Δ → 0.
• The proposed framework is generalizable to other high-frequency financial problems involving time-discretized continuous-time MDPs, such as optimal execution.