[Paper Review] Profitable Strategy Design by Using Deep Reinforcement Learning for Trades on Cryptocurrency Markets

Mohsen Asgari, Seyed Hossein Khasteh|arXiv (Cornell University)|Jan 15, 2022

Financial Markets and Investment Strategies4 citations

TL;DR

This paper proposes a deep reinforcement learning framework using PPO, SAC, and GAIL to design profitable cryptocurrency trading strategies. Trained on price and technical indicator data within a custom Gym environment, the model achieves a 48.5% return (US$4,850 profit on $10,000) over an unseen 66-day test period, with a configurable risk-adjustment hyperparameter in the environment design.

ABSTRACT

Deep Reinforcement Learning solutions have been applied to different control problems with outperforming and promising results. In this research work we have applied Proximal Policy Optimization, Soft Actor-Critic and Generative Adversarial Imitation Learning to strategy design problem of three cryptocurrency markets. Our input data includes price data and technical indicators. We have implemented a Gym environment based on cryptocurrency markets to be used with the algorithms. Our test results on unseen data shows a great potential for this approach in helping investors with an expert system to exploit the market and gain profit. Our highest gain for an unseen 66 day span is 4850 US dollars per 10000 US dollars investment. We also discuss on how a specific hyperparameter in the environment design can be used to adjust risk in the generated strategies.

Motivation & Objective

• To develop an expert trading system using deep reinforcement learning for cryptocurrency markets.
• To evaluate the performance of multiple deep RL algorithms—PPO, SAC, and GAIL—on real-world crypto price data.
• To design a Gym-based reinforcement learning environment tailored for cryptocurrency trading with configurable risk parameters.
• To demonstrate the feasibility of generating profitable, generalizable trading strategies on unseen market data.

Proposed method

• A custom Gym environment was implemented to simulate cryptocurrency trading with state space including price data and technical indicators.
• Proximal Policy Optimization (PPO), Soft Actor-Critic (SAC), and Generative Adversarial Imitation Learning (GAIL) were applied to learn optimal trading policies.
• The state space included historical prices and technical indicators such as RSI and MACD to inform trading decisions.
• The action space allowed for buy, sell, or hold decisions at each time step, with rewards based on portfolio return.
• A risk-adjustment hyperparameter was introduced in the environment to modulate the agent’s risk exposure during training.
• The model was trained on historical data and evaluated on a separate, unseen 66-day test period to assess generalization and profitability.

Experimental results

Research questions

• RQ1 Can deep reinforcement learning agents learn profitable trading strategies on cryptocurrency markets using only price and technical indicator data?
• RQ2 How do PPO, SAC, and GAIL compare in performance and stability when applied to crypto trading strategy generation?
• RQ3 Can a configurable hyperparameter in the environment design effectively control the risk profile of the generated trading strategies?
• RQ4 To what extent can the trained agents generalize to unseen market conditions without overfitting?
• RQ5 What level of profit can be achieved using deep RL-based strategies on real-world crypto data over a test period?

Key findings

• The best-performing agent achieved a 48.5% return, equivalent to $4,850 profit on a $10,000 initial investment over a 66-day unseen test period.
• All three algorithms—PPO, SAC, and GAIL—demonstrated the ability to learn profitable trading policies on cryptocurrency data.
• The inclusion of a configurable risk hyperparameter in the environment allowed for systematic adjustment of risk exposure in the generated strategies.
• The model showed strong generalization performance, maintaining profitability on data not seen during training.
• The results indicate that deep reinforcement learning can be a viable method for constructing expert trading systems in crypto markets.