[Paper Review] A Framework for Empowering Reinforcement Learning Agents with Causal Analysis: Enhancing Automated Cryptocurrency Trading

Rasoul Amirzadeh, Dhananjay Thiruvady+2 more|arXiv (Cornell University)|Oct 14, 2023

Blockchain Technology Applications and SecurityComputer Science3 citations

TL;DR

This paper proposes CausalReinforceNet, a reinforcement learning framework that integrates causal feature selection via Bayesian networks and probabilistic price signals from dynamic Bayesian networks to enhance automated cryptocurrency trading. The framework improves decision-making in volatile altcoin markets, with both PPO and DDPG agents outperforming Buy-and-Hold, achieving strong returns on Binance Coin and Ethereum while maintaining conservative risk management through position sizing limits.

ABSTRACT

Despite advances in artificial intelligence-enhanced trading methods, developing a profitable automated trading system remains challenging in the rapidly evolving cryptocurrency market. This research focuses on developing a reinforcement learning (RL) framework to tackle the complexities of trading five prominent altcoins: Binance Coin, Ethereum, Litecoin, Ripple, and Tether. To this end, we present the CausalReinforceNet~(CRN) framework, which integrates both Bayesian and dynamic Bayesian network techniques to empower the RL agent in trade decision-making. We develop two agents using the framework based on distinct RL algorithms to analyse performance compared to the Buy-and-Hold benchmark strategy and a baseline RL model. The results indicate that our framework surpasses both models in profitability, highlighting CRN's consistent superiority, although the level of effectiveness varies across different cryptocurrencies.

Motivation & Objective

• To address the challenge of developing profitable, automated trading systems in the high-volatility cryptocurrency market, particularly for altcoins.
• To integrate causal feature identification using Bayesian networks to improve state representation in reinforcement learning agents.
• To enhance RL decision-making by incorporating probabilistic price direction signals from dynamic Bayesian networks.
• To design a risk-aware trading system that limits position sizes to mitigate exposure to extreme market fluctuations.
• To evaluate the performance of two distinct RL algorithms—PPO and DDPG—within the CausalReinforceNet framework across multiple altcoins.

Proposed method

• CausalReinforceNet uses Bayesian networks to identify causal features influencing price movements in altcoins, forming the state space for the RL agent.
• Dynamic Bayesian networks generate probabilistic buy/sell signals that guide the RL agent’s action selection, enhancing market signal interpretation.
• The framework enables the RL agent to select both trading actions (buy, sell, hold) and position sizes, integrating risk control into the decision process.
• Two agents are trained using Proximal Policy Optimization (PPO) and Deep Deterministic Policy Gradient (DDPG), allowing comparison of algorithmic performance.
• A conservative trading strategy limits position sizes to reduce exposure, protecting initial capital in volatile markets.
• Feature engineering is tailored per altcoin, acknowledging distinct market behaviors across Binance Coin, Ethereum, Litecoin, Ripple, and Tether.

Figure 1 : RL Classic Cycle – The agent-environment interaction cycle is a continuous loop where an agent interacts with its environment in discrete time steps. At each time step, the agent observes the current state of the environment and selects an action based on its policy. After taking action,

Experimental results

Research questions

• RQ1 How does integrating causal feature selection via Bayesian networks improve the performance of reinforcement learning agents in automated cryptocurrency trading?
• RQ2 To what extent do dynamic Bayesian network predictions enhance the decision-making accuracy of RL agents in volatile altcoin markets?
• RQ3 How do different reinforcement learning algorithms (PPO vs. DDPG) perform when embedded within the CausalReinforceNet framework across multiple altcoins?
• RQ4 What role does conservative position sizing play in mitigating risk while maintaining profitability in high-volatility cryptocurrency trading?
• RQ5 Why do the agents exhibit divergent behaviors on Tether despite consistent DBN predictions, and what does this reveal about asset-specific market dynamics?

Key findings

• Both PPO and DDPG agents outperform the Buy-and-Hold benchmark across all five altcoins, demonstrating the framework’s effectiveness in generating profitable trading strategies.
• The agents achieved notable returns on investment for Binance Coin and Ethereum, with positive average ROIs across both algorithms.
• The DDPG agent exhibited a conservative strategy, performing more hold actions despite higher DBN-generated buy signals, indicating caution in uncertain conditions.
• For Tether, the DDPG agent executed 53.45% buy actions despite a 59.87% DBN prediction of a downward price movement, suggesting a contrarian or risk-on approach specific to this stablecoin.
• The agents showed divergent performance on Ripple, with one agent generating positive returns and the other negative, highlighting the sensitivity of RL agents to asset-specific dynamics.
• Hold actions were the most frequent strategy, indicating that market uncertainty and lack of clear signals led the agents to avoid trading in many periods.

Figure 2 : CausalReinforceNet Architecture – The framework comprises four essential elements. The first category encompasses input features containing market data, technical indicators, financial assets, and social media. The second element consists of a BN module, which is responsible for the featu