[Paper Review] Predicting The Stock Trend Using News Sentiment Analysis and Technical Indicators in Spark

Taylan Kabbani, Fatih Enes Usta|arXiv (Cornell University)|Jan 19, 2022

Stock Market Forecasting Methods18 citations

TL;DR

This paper proposes a hybrid stock trend prediction model using Spark to combine technical indicators and news sentiment scores for binary classification (uptrend/downtrend). It applies Logistic Regression, Random Forest, and Gradient Boosting Machine on S&P 500 stocks (AAPL, AMZN, NFLX), achieving 63.58% test accuracy with Random Forest as the top-performing model.

ABSTRACT

Predicting the stock market trend has always been challenging since its movement is affected by many factors. Here, we approach the future trend prediction problem as a machine learning classification problem by creating tomorrow_trend feature as our label to be predicted. Different features are given to help the machine learning model predict the label of a given day; whether it is an uptrend or downtrend, those features are technical indicators generated from the stock's price history. In addition, as financial news plays a vital role in changing the investor's behavior, the overall sentiment score on a given day is created from all news released on that day and added to the model as another feature. Three different machine learning models are tested in Spark (big-data computing platform), Logistic Regression, Random Forest, and Gradient Boosting Machine. Random Forest was the best performing model with a 63.58% test accuracy.

Motivation & Objective

• To improve stock trend prediction accuracy by integrating technical indicators and financial news sentiment using big data platforms.
• To evaluate the effectiveness of combining fundamental (news sentiment) and technical (price-based indicators) analysis in a scalable machine learning pipeline.
• To assess the performance of multiple machine learning models—Logistic Regression, Random Forest, and Gradient Boosting Machine—on stock trend classification using Spark.
• To reduce training time and enhance hyperparameter tuning through parallelized execution in Apache Spark's MLlib framework.

Proposed method

• Collected daily stock price data (Open, High, Low, Close, Volume, Adjusted Close) and computed 10 technical indicators, including SMA, RSI, MACD, and Bollinger Bands.
• Gathered 9.2 GB of financial news articles from 2016–2020 and processed them using BERT-based NLP models to extract daily sentiment scores via mean-pooled sentence embeddings.
• Aggregated daily sentiment scores by grouping all news released on the same day and computed an overall sentiment score per trading day.
• Constructed a final feature set of 9 variables per stock: 6 technical indicators (SMA, RSI, MACD, Bollinger Bands, OBV, Stochastic Oscillator), volume, and the aggregated daily sentiment score.
• Applied a Spark ML pipeline using MinMaxScaler for feature normalization, RFormula for one-hot encoding of categorical labels, and 10-fold cross-validation with grid search for hyperparameter tuning.
• Trained and evaluated three classifiers—Logistic Regression, Random Forest, and Gradient Boosting Machine—on 80% training and 20% test splits, using accuracy, precision, recall, and F1-score as evaluation metrics.

Experimental results

Research questions

• RQ1 Can combining technical indicators with news sentiment scores improve stock trend prediction accuracy compared to using price data alone?
• RQ2 How do different machine learning models (Logistic Regression, Random Forest, Gradient Boosting Machine) perform in classifying stock trend direction (uptrend/downtrend) when trained on hybrid financial and sentiment features?
• RQ3 To what extent does using Apache Spark enhance scalability and training efficiency for stock prediction models on large-scale financial datasets?
• RQ4 What is the impact of feature correlation on model performance, and how does feature selection (e.g., retaining only non-reduundant features) affect prediction accuracy?

Key findings

• Random Forest achieved the highest test accuracy of 63.58%, outperforming Logistic Regression and Gradient Boosting Machine across all three stocks (AAPL, AMZN, NFLX).
• The model demonstrated a training accuracy of 65.5%, indicating moderate generalization with slight overfitting, but maintained strong performance on the independent test set.
• Feature correlation analysis revealed high interdependence among price-based features (Open, High, Low, Close, Adj Close, SMA), leading to the retention of only SMA, High, and Close to reduce redundancy.
• The use of BERT-based NLP for sentiment analysis enabled context-aware sentiment scoring from financial news, which was successfully integrated as a predictive feature.
• The Spark-based pipeline enabled efficient parallel processing, significantly reducing training time and supporting extensive hyperparameter tuning across a large grid of configurations.
• Despite the model’s moderate accuracy, the results suggest that integrating sentiment from news and technical indicators can yield meaningful predictive signals in volatile markets.