[Paper Review] Reinforcement Learning for Market Making in a Multi-agent Dealer Market

Sumitra Ganesh, Nelson Vadori+4 more|arXiv (Cornell University)|Nov 14, 2019

Auction Theory and ApplicationsDecision Sciences16 references19 citations

TL;DR

This paper proposes a multi-agent reinforcement learning (RL) framework for simulating and training market makers in a dealer market environment. Using a custom simulator, the RL agent learns to adaptively set bid-ask spreads, skew prices based on inventory, and dynamically adjust position size according to market price drift, achieving robust performance across competitive and risk-sensitive scenarios.

ABSTRACT

Market makers play an important role in providing liquidity to markets by continuously quoting prices at which they are willing to buy and sell, and managing inventory risk. In this paper, we build a multi-agent simulation of a dealer market and demonstrate that it can be used to understand the behavior of a reinforcement learning (RL) based market maker agent. We use the simulator to train an RL-based market maker agent with different competitive scenarios, reward formulations and market price trends (drifts). We show that the reinforcement learning agent is able to learn about its competitor's pricing policy; it also learns to manage inventory by smartly selecting asymmetric prices on the buy and sell sides (skewing), and maintaining a positive (or negative) inventory depending on whether the market price drift is positive (or negative). Finally, we propose and test reward formulations for creating risk averse RL-based market maker agents.

Motivation & Objective

• To develop a realistic multi-agent simulation environment for studying reinforcement learning-based market making in dealer (over-the-counter) markets.
• To investigate how RL agents learn competitive pricing strategies and inventory management under partial observability.
• To evaluate the impact of different reward formulations on risk aversion and inventory control in RL-based market makers.
• To understand how market price drift influences the RL agent's optimal inventory positioning and skewing behavior.
• To design and test risk penalty functions that produce more stable, risk-averse market-making policies.

Proposed method

• Formalize a dealer market as a multi-agent system with interacting market makers and investor agents, where only direct transaction data is observable.
• Implement an adaptive market maker agent based on mean-variance optimization as a realistic competitor for the RL agent.
• Train the RL agent using deep Q-networks (DQN) with state representations including inventory, mid-price, and order flow.
• Design reward functions that include inventory risk penalties (e.g., quadratic and exponential penalties) to encourage risk-averse behavior.
• Simulate various competitive scenarios: persistent, random, and adaptive competitors with different pricing policies.
• Use a time-discretized environment with 200 timesteps per rollout to study long-term behavior and convergence.

Experimental results

Research questions

• RQ1 Can an RL-based market maker learn the pricing policy of a competing market maker through partial observations of trades?
• RQ2 How does the RL agent adapt its bid-ask spread and price skewing in response to inventory levels and market drift?
• RQ3 To what extent can reward shaping with risk penalties reduce inventory volatility and stabilize position size?
• RQ4 How does the market price drift influence the optimal inventory position of the RL agent?
• RQ5 Can the RL agent learn to exploit positive or negative drift by maintaining a directional inventory position?

Key findings

• The RL agent successfully learns to infer the pricing strategy of a competing adaptive market maker by observing only its own trades and execution outcomes.
• The RL agent autonomously develops price skewing behavior—lowering bid spreads when inventory is negative and sell spreads when inventory is positive—without explicit bias or prior instruction.
• Under positive drift (μ = 1), the RL agent learns to build a large positive inventory (mean = 16.06 over 500 timesteps), while under negative drift (μ = -1), it holds a negative inventory (mean = -16.06), demonstrating dynamic adaptation to market trends.
• The inclusion of risk penalty functions in the reward signal reduces the standard deviation of inventory and inventory PnL, and decreases the absolute mean inventory, confirming the effectiveness of risk-averse design.
• The RL agent achieves convergence in policy learning across multiple training runs, with stable performance against both persistent and adaptive competitors.
• The agent's performance is robust across different market conditions, showing that it generalizes well to varying levels of drift and competitor behavior.