[Paper Review] Reinforced Deep Markov Models With Applications in Automatic Trading

Tadeu Augusto Ferreira|arXiv (Cornell University)|Nov 9, 2020

Stock Market Forecasting Methods27 references4 citations

TL;DR

This paper proposes the Reinforced Deep Markov Model (RDMM), a model-based reinforcement learning framework that integrates variational autoencoders and deep Markov models to handle noisy, incomplete market data in algorithmic trading. By filtering observations and incorporating state uncertainty into policy learning, RDMM improves data efficiency and financial performance, outperforming Q-learning, DynaQ-ARIMA, and DynaQ-LSTM benchmarks on real limit order book data from Facebook, Intel, Vodafone, and Microsoft.

ABSTRACT

Inspired by the developments in deep generative models, we propose a model-based RL approach, coined Reinforced Deep Markov Model (RDMM), designed to integrate desirable properties of a reinforcement learning algorithm acting as an automatic trading system. The network architecture allows for the possibility that market dynamics are partially visible and are potentially modified by the agent's actions. The RDMM filters incomplete and noisy data, to create better-behaved input data for RL planning. The policy search optimisation also properly accounts for state uncertainty. Due to the complexity of the RKDF model architecture, we performed ablation studies to understand the contributions of individual components of the approach better. To test the financial performance of the RDMM we implement policies using variants of Q-Learning, DynaQ-ARIMA and DynaQ-LSTM algorithms. The experiments show that the RDMM is data-efficient and provides financial gains compared to the benchmarks in the optimal execution problem. The performance improvement becomes more pronounced when price dynamics are more complex, and this has been demonstrated using real data sets from the limit order book of Facebook, Intel, Vodafone and Microsoft.

Motivation & Objective

• To address the limitations of classical Q-learning in high-dimensional, noisy financial environments with partial observability.
• To improve data efficiency and robustness in algorithmic trading by integrating uncertainty modeling into the policy learning process.
• To develop a unified framework that combines deep generative models with model-based reinforcement learning for financial time series.
• To evaluate the performance of the proposed RDMM against established benchmarks in optimal execution tasks using real market data.

Proposed method

• The RDMM employs a deep Markov model architecture enhanced with variational inference to model latent states from noisy, incomplete market observations.
• It uses a differentiable filtering mechanism to estimate the posterior distribution over latent states, accounting for both observation noise and model uncertainty.
• The policy is optimized using a gradient-based approach that maximizes expected return while explicitly incorporating uncertainty in the filtered state estimates.
• The framework integrates model-based planning via a learned dynamics model, enabling efficient experience replay and simulation-based learning.
• The model is trained end-to-end using backpropagation through time and differentiable reinforcement learning objectives, with ablation studies to isolate component contributions.
• The approach is evaluated using Q-learning, DynaQ-ARIMA, and DynaQ-LSTM as baselines, with the RDMM replacing the model component in DynaQ.

Experimental results

Research questions

• RQ1 Can a deep generative model improve data efficiency and performance in reinforcement learning-based algorithmic trading under partial observability?
• RQ2 How does incorporating state uncertainty into policy optimization affect financial performance in optimal execution tasks?
• RQ3 Does the RDMM outperform standard model-free and model-based baselines in real market data with complex price dynamics?
• RQ4 What is the contribution of each component—VAE, DMM, and uncertainty-aware policy—on overall performance?

Key findings

• The RDMM achieves significant financial gains compared to Q-learning, DynaQ-ARIMA, and DynaQ-LSTM across all tested stocks, with performance improvements most pronounced in complex market regimes.
• The model demonstrates superior data efficiency, learning effective policies with fewer interactions due to accurate state filtering and uncertainty-aware planning.
• Ablation studies confirm that the integration of variational inference and uncertainty modeling is critical for performance, especially under noisy or incomplete observations.
• On real limit order book data from Facebook, Intel, Vodafone, and Microsoft, the RDMM consistently outperformed all baselines in terms of cumulative profit and risk-adjusted returns.
• The performance gain increases with market complexity, indicating the model’s robustness in volatile and non-Markovian environments.