[Paper Review] Enhancing Price Prediction in Cryptocurrency Using Transformer Neural Network and Technical Indicators

Mohammad Ali Labbaf Khaniki, Mohammad Manthouri|arXiv (Cornell University)|Mar 6, 2024

Blockchain Technology Applications and Security4 citations

TL;DR

This paper proposes a hybrid deep learning model combining Performer-based Transformers, BiLSTM, and technical indicators to enhance Bitcoin, Ethereum, and Litecoin price prediction. By leveraging FAVOR+ attention for computational efficiency and bidirectional sequence modeling, the approach achieves superior performance on hourly and daily timeframes compared to existing methods.

ABSTRACT

This study presents an innovative approach for predicting cryptocurrency time series, specifically focusing on Bitcoin, Ethereum, and Litecoin. The methodology integrates the use of technical indicators, a Performer neural network, and BiLSTM (Bidirectional Long Short-Term Memory) to capture temporal dynamics and extract significant features from raw cryptocurrency data. The application of technical indicators, such facilitates the extraction of intricate patterns, momentum, volatility, and trends. The Performer neural network, employing Fast Attention Via positive Orthogonal Random features (FAVOR+), has demonstrated superior computational efficiency and scalability compared to the traditional Multi-head attention mechanism in Transformer models. Additionally, the integration of BiLSTM in the feedforward network enhances the model's capacity to capture temporal dynamics in the data, processing it in both forward and backward directions. This is particularly advantageous for time series data where past and future data points can influence the current state. The proposed method has been applied to the hourly and daily timeframes of the major cryptocurrencies and its performance has been benchmarked against other methods documented in the literature. The results underscore the potential of the proposed method to outperform existing models, marking a significant progression in the field of cryptocurrency price prediction.

Motivation & Objective

• To improve the accuracy of cryptocurrency price prediction using advanced deep learning architectures.
• To address the limitations of standard attention mechanisms in Transformers by employing FAVOR+ for better computational scalability.
• To integrate technical indicators to extract momentum, volatility, and trend patterns from raw price data.
• To enhance temporal feature learning by combining bidirectional LSTM with attention-based modeling for improved sequence representation.
• To benchmark the proposed model against existing state-of-the-art methods on major cryptocurrencies across multiple timeframes.

Proposed method

• The model uses technical indicators (e.g., RSI, MACD, Bollinger Bands) as input features to encode market regime characteristics.
• A Performer neural network with FAVOR+ attention replaces standard multi-head attention to reduce computational complexity while maintaining long-range dependency modeling.
• The Performer encoder is followed by a BiLSTM layer to capture bidirectional temporal dependencies in price sequences.
• The fusion of technical indicators and raw price data is processed through the hybrid architecture to generate multi-step price forecasts.
• The model is trained end-to-end on hourly and daily price data for Bitcoin, Ethereum, and Litecoin using standard regression loss functions.
• Performance is evaluated using standard metrics such as RMSE, MAE, and R² on test splits, with comparisons to baseline models from the literature.

Experimental results

Research questions

• RQ1 Can a Performer-based Transformer with FAVOR+ attention outperform standard Transformers in cryptocurrency price prediction tasks?
• RQ2 How does the integration of technical indicators improve the predictive performance of deep learning models on cryptocurrency time series?
• RQ3 To what extent does the inclusion of BiLSTM enhance temporal feature learning compared to unidirectional or self-attention-only architectures?
• RQ4 Does the proposed hybrid model generalize across different cryptocurrencies and timeframes (hourly vs. daily)?
• RQ5 How does the model compare quantitatively to existing state-of-the-art approaches in terms of prediction accuracy and computational efficiency?

Key findings

• The proposed model achieved lower RMSE and MAE values than baseline models on both hourly and daily price prediction tasks for Bitcoin, Ethereum, and Litecoin.
• The use of FAVOR+ attention enabled faster inference and reduced memory consumption compared to standard multi-head attention, improving scalability on long sequences.
• Incorporating technical indicators significantly enhanced model performance, particularly in capturing trend reversals and volatility regimes.
• The BiLSTM component contributed to improved learning of long-term dependencies by processing sequences in both forward and backward directions.
• The model outperformed existing state-of-the-art approaches in terms of predictive accuracy, as validated through benchmark comparisons on public datasets.
• The ablation study confirmed that the combination of technical indicators, Performer, and BiLSTM yielded the highest predictive performance, demonstrating the synergy of components.