[Paper Review] Support for Stock Trend Prediction Using Transformers and Sentiment Analysis

Harsimrat Kaeley, Ye Qiao+1 more|arXiv (Cornell University)|May 18, 2023

Stock Market Forecasting MethodsDecision Sciences3 citations

TL;DR

This paper proposes StockFormer, a Transformer-based model that combines technical stock data and sentiment analysis from news headlines to improve long-term stock trend prediction. By leveraging self-attention to capture long-term dependencies and integrating real-time sentiment signals, the model achieves up to 18.63% higher directional accuracy than RNNs at 30-day prediction windows, demonstrating superior performance on extended time horizons.

ABSTRACT

Stock trend analysis has been an influential time-series prediction topic due to its lucrative and inherently chaotic nature. Many models looking to accurately predict the trend of stocks have been based on Recurrent Neural Networks (RNNs). However, due to the limitations of RNNs, such as gradient vanish and long-term dependencies being lost as sequence length increases, in this paper we develop a Transformer based model that uses technical stock data and sentiment analysis to conduct accurate stock trend prediction over long time windows. This paper also introduces a novel dataset containing daily technical stock data and top news headline data spanning almost three years. Stock prediction based solely on technical data can suffer from lag caused by the inability of stock indicators to effectively factor in breaking market news. The use of sentiment analysis on top headlines can help account for unforeseen shifts in market conditions caused by news coverage. We measure the performance of our model against RNNs over sequence lengths spanning 5 business days to 30 business days to mimic different length trading strategies. This reveals an improvement in directional accuracy over RNNs as sequence length is increased, with the largest improvement being close to 18.63% at 30 business days.

Motivation & Objective

• To address the limitations of RNNs in capturing long-term dependencies and handling market news lags in stock trend prediction.
• To develop a Transformer-based model that integrates technical indicators and real-time sentiment from news headlines for improved forecasting accuracy.
• To evaluate the model’s performance across varying sequence lengths (5 to 30 business days) to assess scalability for different trading strategies.
• To introduce a novel dataset with daily technical data and top news headlines spanning nearly three years for long-term financial forecasting research.
• To compare the proposed model’s performance against RNNs and recent Transformer-based models, particularly on extended time windows not previously explored.

Proposed method

• The model uses a Transformer encoder architecture with multi-head self-attention to model long-range temporal dependencies across input sequences of stock data and headlines.
• Technical stock features (e.g., open, high, low, close, volume) are normalized and fed into the model alongside embedded sentiment scores derived from top news headlines.
• Sentiment analysis is applied to daily news headlines using NLP techniques to generate real-time sentiment embeddings that are concatenated with technical features.
• The model is trained to predict the normalized opening price of the next day (n+1) based on an n-day lag window of data, framed as a regression task.
• Positional encoding using a sinusoidal function is applied to preserve temporal order in the input sequence, enabling the model to learn temporal dynamics.
• Performance is evaluated using directional accuracy, mean squared error (MSE), and R², with comparisons made against LSTM, GRU, and S_EMDAM_LSTM baselines across multiple window sizes.

Experimental results

Research questions

• RQ1 Can a Transformer-based model outperform RNNs in long-term stock trend prediction by better capturing long-term dependencies?
• RQ2 To what extent does incorporating sentiment analysis from news headlines improve directional accuracy compared to technical analysis alone?
• RQ3 How does model performance vary across different sequence lengths (5 to 30 business days), and does the performance gap between Transformers and RNNs increase with longer horizons?
• RQ4 Does the integration of real-time sentiment signals mitigate the lag inherent in technical indicators during market shifts?
• RQ5 How does the proposed StockFormer model compare to recent Transformer-based models like TEANet, especially in terms of accuracy on similar short- and long-term windows?

Key findings

• The StockFormer model achieves a directional accuracy of 89.19% at a 30-day prediction window, representing a peak improvement of 18.63% over RNN-based models.
• Directional accuracy increases with sequence length, demonstrating that the Transformer’s self-attention mechanism effectively captures long-term dependencies that RNNs fail to maintain.
• The model outperforms Bi-Directional LSTM, GRU-News, and S_EMDAM_LSTM baselines at longer horizons, particularly at 30-day windows where it significantly exceeds RNN-based models.
• At a 5-day window, StockFormer achieves 60.36% directional accuracy, slightly below TEANet’s 65.16%, indicating that RNN-inclusive models may have an edge in short-term forecasting.
• The integration of sentiment analysis from headlines improves model responsiveness to market shocks, reducing the disconnect between lagging technical indicators and real-time market movements.
• The model’s performance is robust across multiple metrics, with an R² of 0.9219 and MSE of 0.1071 at the 30-day window, indicating strong regression performance on normalized price trends.