[Paper Review] Extending Deep Reinforcement Learning Frameworks in Cryptocurrency Market Making

Jonathan Sadighian|arXiv (Cornell University)|Apr 15, 2020

Stock Market Forecasting Methods11 references4 citations

TL;DR

This paper proposes a price-based event-driven reinforcement learning framework for cryptocurrency market making, replacing traditional time- or tick-based events with price-change triggers. Using A2C and PPO algorithms on Bitmex LOB data, the approach achieves improved profit and stability—especially with the TC reward function—outperforming time-based baselines across seven reward functions and 30-day out-of-sample testing.

ABSTRACT

There has been a recent surge in interest in the application of artificial intelligence to automated trading. Reinforcement learning has been applied to single- and multi-instrument use cases, such as market making or portfolio management. This paper proposes a new approach to framing cryptocurrency market making as a reinforcement learning challenge by introducing an event-based environment wherein an event is defined as a change in price greater or less than a given threshold, as opposed to by tick or time-based events (e.g., every minute, hour, day, etc.). Two policy-based agents are trained to learn a market making trading strategy using eight days of training data and evaluate their performance using 30 days of testing data. Limit order book data recorded from Bitmex exchange is used to validate this approach, which demonstrates improved profit and stability compared to a time-based approach for both agents when using a simple multi-layer perceptron neural network for function approximation and seven different reward functions.

Motivation & Objective

• To address the limitations of latency-sensitive, time- or tick-based event environments in reinforcement learning for market making.
• To propose and evaluate a novel price-based event environment where agents act only upon significant price changes.
• To compare the performance of multiple reward functions in a deep reinforcement learning framework for automated market making.
• To extend the DRLMM framework with a more stable, intelligent strategy that reduces noise from high-frequency LOB data.

Proposed method

• The framework uses a Markov Decision Process (MDP) with state space derived from Level II LOB data, including order flow imbalances and derived indicators.
• Agents are trained using policy-based deep reinforcement learning algorithms: A2C and PPO, with a multi-layer perceptron for function approximation.
• The environment is defined by price-based events—actions are triggered only when price changes exceed a threshold—reducing noise from non-significant ticks.
• Seven distinct reward functions are evaluated: UPnL, UPnLwF, Asym, AsymC, ΔRPnL, TC, and DSR, each shaping agent behavior toward different risk-return objectives.
• Training uses eight days of Bitmex LOB data (1M steps), with evaluation on 30 out-of-sample days to assess generalization and profitability.
• Performance is measured via cumulative return, profit-and-loss, and stability metrics across multiple experiment sets.

Experimental results

Research questions

• RQ1 How does a price-based event environment compare to time- or tick-based environments in terms of agent profitability and stability in cryptocurrency market making?
• RQ2 Which of seven reward functions yields the most robust and profitable trading strategies under both time- and price-based event settings?
• RQ3 Can policy-based deep reinforcement learning agents generalize across different market regimes when trained on limited historical data?
• RQ4 Does reducing event frequency via price thresholds improve learning efficiency and reduce erratic trading behavior in high-frequency LOB environments?

Key findings

• The A2C agent outperformed PPO in cumulative return and number of profitable experiments across both event types.
• The TC (Target Completion) reward function generated the highest return for both time- and price-based environments, with a 17.61% return in Set 3 under time-based events.
• Price-based events led to more stable trading patterns, especially during large price jumps, and reduced noise-induced erratic behavior.
• 23 out of 84 experiments using price-based events were profitable, compared to fewer under time-based environments, indicating better learning efficiency.
• The AsymC reward function showed strong performance in some sets, achieving 11.88% return in one price-based experiment, suggesting asymmetric risk adjustment can be effective.
• The DSR (Differential Sharpe Ratio) reward function yielded mixed results, with negative returns in most cases, indicating potential instability in high-volatility regimes.