[Paper Review] Using NLP on news headlines to predict index trends

Marc Velay, Fabrice Daniel|arXiv (Cornell University)|Jun 22, 2018

Advanced Text Analysis Techniques2 references3 citations

TL;DR

This study investigates whether natural language processing (NLP) of news headlines can predict daily trends in the Dow Jones Industrial Average (DJIA). Using sentiment analysis, word embeddings (Word2Vec, TF-IDF, Bag-of-Words), and machine learning models (LSTM, MLP, SVM), the authors find that predictive performance is only slightly better than random chance, with top accuracy at 57% for logistic regression, suggesting limited correlation between headline sentiment and index movements over long timeframes.

ABSTRACT

This paper attempts to provide a state of the art in trend prediction using news headlines. We present the research done on predicting DJIA trends using Natural Language Processing. We will explain the different algorithms we have used as well as the various embedding techniques attempted. We rely on statistical and deep learning models in order to extract information from the corpuses.

Motivation & Objective

• To determine if news headline sentiment and linguistic features can predict daily DJIA index movements.
• To evaluate the effectiveness of various NLP techniques—such as sentiment analysis, word embeddings, and named entity recognition—in forecasting financial trends.
• To compare traditional machine learning models (e.g., SVM, Random Forest) and deep learning models (e.g., LSTM, MLP) for trend prediction using textual data.
• To assess whether combining sentiment, subjectivity, and financial indicators improves predictive accuracy.

Proposed method

• The study uses a corpus of the top 25 news headlines per business day from 2008 to late 2015, covering the DJIA.
• Text preprocessing includes stop-word removal, named entity recognition (NER), and punctuation removal to refine input data.
• Word embeddings are generated using Word2Vec, TF-IDF, and Bag-of-Words, with and without N-grams to enrich feature representation.
• Sentiment and subjectivity scores are extracted using a lexicon-based approach, with attempts to detect sarcasm.
• Multiple classifiers—including Logistic Regression, SVM, Random Forest, Extreme Gradient Boosting, LSTM, and MLP—are trained and evaluated on the encoded headline features.
• Model evaluation uses 9-month training and 3-month testing windows to ensure temporal relevance, avoiding data leakage.

Experimental results

Research questions

• RQ1 Can NLP-extracted sentiment from news headlines predict daily movements in the DJIA index with statistically significant accuracy?
• RQ2 How do different word embedding techniques (Word2Vec, TF-IDF, Bag-of-Words) compare in predicting index trends when combined with machine learning models?
• RQ3 Does combining sentiment, subjectivity, and financial indicators improve predictive performance over using sentiment alone?
• RQ4 To what extent does the temporal distribution of news data affect model generalization and long-term predictive validity?
• RQ5 Can multimodal learning (e.g., sentiment + context + financial data) yield better results than unimodal approaches?

Key findings

• The highest validation accuracy achieved was 57% using Logistic Regression, indicating only a marginal improvement over random chance (50%).
• LSTM models achieved 55% accuracy, suggesting recurrent networks offer slightly better temporal modeling than feedforward networks.
• Combining sentiment, subjectivity, and financial indicators did not significantly improve performance, with results only marginally better than baseline.
• Word2Vec embeddings performed best when using sum of vectors rather than mean or distance-based aggregation, improving performance by 4–5 percentage points.
• The model's predictive power is limited by data bias, including negativity bias in news media and temporal irrelevance of long-term training on evolving language and events.
• The study concludes that current NLP models cannot reliably predict index trends from headlines due to weak correlation and model instability over time.