[Paper Review] Double Deep Q-Learning for Optimal Execution

Brian Ning, Franco Ho Ting Ling+1 more|arXiv (Cornell University)|Dec 17, 2018

Stock Market Forecasting Methods13 references22 citations

TL;DR

This paper proposes a Double Deep Q-Learning (DDQN) framework for optimal trade execution, using a fully connected neural network to estimate Q-values from limit order book features, trading signals, and action space. The method outperforms the TWAP benchmark on seven of nine stocks, with higher median and mean outperformance, improved gain-loss ratios, and over 95% probability of outperforming TWAP for several assets.

ABSTRACT

Optimal trade execution is an important problem faced by essentially all traders. Much research into optimal execution uses stringent model assumptions and applies continuous time stochastic control to solve them. Here, we instead take a model free approach and develop a variation of Deep Q-Learning to estimate the optimal actions of a trader. The model is a fully connected Neural Network trained using Experience Replay and Double DQN with input features given by the current state of the limit order book, other trading signals, and available execution actions, while the output is the Q-value function estimating the future rewards under an arbitrary action. We apply our model to nine different stocks and find that it outperforms the standard benchmark approach on most stocks using the measures of (i) mean and median out-performance, (ii) probability of out-performance, and (iii) gain-loss ratios.

Motivation & Objective

• To develop a model-free reinforcement learning approach for optimal execution that avoids restrictive parametric assumptions.
• To adapt Deep Q-Learning with Double DQN and experience replay to handle the continuous state and action space of limit order books.
• To evaluate performance against the standard TWAP benchmark using real market data across multiple equities.
• To assess the impact of including quadratic variation (QV) as a volatility proxy in the state representation.
• To provide a financially interpretable policy that generalizes across different market regimes and stock characteristics.

Proposed method

• A fully connected deep neural network is trained to estimate the Q-value function mapping state-action pairs to expected cumulative rewards.
• The state representation includes time, inventory, mid-price, bid-ask spread, and quadratic variation (QV) as features.
• Double DQN is used to reduce overestimation bias in Q-value estimation, improving learning stability.
• Experience replay is employed to break temporal correlation and improve data efficiency during training.
• The action space consists of discrete execution sizes, with the agent selecting the action maximizing estimated Q-value at each step.
• The network is trained using a replay buffer storing (state, action, reward, next_state) transitions, with target network updates for stability.

Experimental results

Research questions

• RQ1 Can a deep reinforcement learning approach outperform the standard TWAP benchmark in optimal execution without relying on parametric price models?
• RQ2 How does the inclusion of quadratic variation (QV) as a volatility feature affect the learned trading policy and performance?
• RQ3 To what extent does the DDQN model generalize across different stocks with varying liquidity and volatility?
• RQ4 How do the learned policies reflect known economic intuition, such as increasing execution speed as inventory increases or as time to horizon decreases?
• RQ5 What is the impact of state representation (e.g., time, price, inventory, QV) on the robustness and outperformance of the strategy?

Key findings

• The DDQN-based strategy outperforms TWAP on seven out of nine tested stocks, with statistically significant outperformance in median and mean relative P&L.
• For INTC, the median outperformance reaches 11.63 basis points (TIP) and 11.96 basis points (TIPQV), with a 95.8% and 97.8% probability of outperforming TWAP, respectively.
• For VOD, the median outperformance is 14.68 basis points (TIP) and 15.72 basis points (TIPQV), with a 97.8% and 99.2% probability of outperforming TWAP, respectively.
• The inclusion of QV as a feature generally improves performance, as seen in higher median outperformance and gain-loss ratios, especially in high-volatility regimes.
• The strategy exhibits economically intuitive behavior: execution increases with time, price, and volatility (QV), and decreases with remaining inventory.
• Only Amazon and Google show no significant improvement over TWAP, with performance metrics near or below zero, suggesting limited benefit in low-liquidity or high-noise environments.