[Paper Review] A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem

Zhengyao Jiang, Dixing Xu+1 more|arXiv (Cornell University)|Jun 30, 2017

Blockchain Technology Applications and Security229 citations

TL;DR

The paper introduces a model-free deep reinforcement learning framework (EIIE) for multi-asset portfolio management, with online learning and an explicit reward, and shows top performance on cryptocurrency back-tests using CNN, RNN, and LSTM implementations.

ABSTRACT

Financial portfolio management is the process of constant redistribution of a fund into different financial products. This paper presents a financial-model-free Reinforcement Learning framework to provide a deep machine learning solution to the portfolio management problem. The framework consists of the Ensemble of Identical Independent Evaluators (EIIE) topology, a Portfolio-Vector Memory (PVM), an Online Stochastic Batch Learning (OSBL) scheme, and a fully exploiting and explicit reward function. This framework is realized in three instants in this work with a Convolutional Neural Network (CNN), a basic Recurrent Neural Network (RNN), and a Long Short-Term Memory (LSTM). They are, along with a number of recently reviewed or published portfolio-selection strategies, examined in three back-test experiments with a trading period of 30 minutes in a cryptocurrency market. Cryptocurrencies are electronic and decentralized alternatives to government-issued money, with Bitcoin as the best-known example of a cryptocurrency. All three instances of the framework monopolize the top three positions in all experiments, outdistancing other compared trading algorithms. Although with a high commission rate of 0.25% in the backtests, the framework is able to achieve at least 4-fold returns in 50 days.

Motivation & Objective

• Provide a model-free RL solution to the portfolio management problem that avoids price prediction.
• Incorporate transaction costs and past portfolio information into the learning framework.
• Develop a scalable architecture (EIIE) that can handle multiple assets and online learning across markets.
• Evaluate the framework against existing strategies in back-tests on cryptocurrency data.

Proposed method

• Propose Ensemble of Identical Independent Evaluators (EIIE) topology to produce portfolio weights via a softmax over asset evaluations.
• Introduce Portfolio-Vector Memory (PVM) to store previous period weights for transaction-cost awareness.
• Use Online Stochastic Batch Learning (OSBL) to train the EIIE in both pre-trade and online settings.
• Define an explicit objective reward as the average of periodic logarithmic returns to guide gradient ascent.
• Experiment with three model variants: CNN, basic RNN, and LSTM as the EIIE core.
• Preselect the top-volume 11 non-cash assets (plus cash) for a 12-asset portfolio and normalize inputs through a price tensor X_t that includes closing, high, and low prices.
• Handle missing data by filling NANs with flat price movements to avoid leakage of asset identity and preserve generalization.

Experimental results

Research questions

• RQ1 Can a model-free deep RL framework with an EIIE topology outperform traditional portfolio strategies in a multi-asset setting?
• RQ2 How does incorporating transaction costs via a transaction remainder factor mu_t affect learning and performance?
• RQ3 Does using a continuous action policy (via softmax over evaluator outputs) provide advantages over discretized actions in portfolio management?
• RQ4 Is the framework robust when applied to highly volatile, open, continuous markets such as cryptocurrencies?

Key findings

• The EIIEs monopolize the top three positions in all three back-test experiments, outperforming other strategies.
• With a back-test trading period of 30 minutes, the framework achieves at least 4-fold returns in 50 days despite a 0.25% commission.
• Three network variants (CNN, basic RNN, and LSTM) are evaluated and all are competitive, with the ensemble framework providing superior performance.
• The framework is evaluated on a cryptocurrency exchange (Poloniex) with preselected assets and explicit transaction-cost modeling, showing strong profitability under the proposed RL setup.
• The reward function is explicit and based on the average of periodic logarithmic returns, guiding the EIIE to optimize wealth over time.