[Paper Review] Assessing Look-Ahead Bias in Stock Return Predictions Generated By GPT Sentiment Analysis

Paul Glasserman, Caden Lin|arXiv (Cornell University)|Sep 29, 2023

Stock Market Forecasting Methods11 citations

TL;DR

The paper exposes look-ahead bias and distraction effects in GPT-based sentiment trading signals from financial headlines, showing anonymization of company names reduces in-sample bias and may improve out-of-sample performance, especially for larger firms.

ABSTRACT

Large language models (LLMs), including ChatGPT, can extract profitable trading signals from the sentiment in news text. However, backtesting such strategies poses a challenge because LLMs are trained on many years of data, and backtesting produces biased results if the training and backtesting periods overlap. This bias can take two forms: a look-ahead bias, in which the LLM may have specific knowledge of the stock returns that followed a news article, and a distraction effect, in which general knowledge of the companies named interferes with the measurement of a text's sentiment. We investigate these sources of bias through trading strategies driven by the sentiment of financial news headlines. We compare trading performance based on the original headlines with de-biased strategies in which we remove the relevant company's identifiers from the text. In-sample (within the LLM training window), we find, surprisingly, that the anonymized headlines outperform, indicating that the distraction effect has a greater impact than look-ahead bias. This tendency is particularly strong for larger companies--companies about which we expect an LLM to have greater general knowledge. Out-of-sample, look-ahead bias is not a concern but distraction remains possible. Our proposed anonymization procedure is therefore potentially useful in out-of-sample implementation, as well as for de-biased backtesting.

Motivation & Objective

• Motivate the problem of look-ahead bias in LLM-driven trading using financial news sentiment.
• Develop an anonymization method to remove entity identifiers from headlines.
• Compare in-sample and out-of-sample trading performance of original vs. anonymized headlines.
• Assess how company size affects bias and predictive power of GPT-based scores.
• Provide practical de-biasing guidance for future LLM backtesting.

Proposed method

• Use GPT-3.5-Turbo with a fixed prompt to classify headlines as good/neutral/bad for stock prices.
• Compute daily trading signals from GPT scores and evaluate long-only, short-only, and long-short strategies.
• Anonymize headlines by replacing company identifiers and relevant products/services using fuzzy matching and knowledge graph enhancements.
• Compare in-sample and out-of-sample performance across scraped and Thomson Reuters headline sets.
• Analyze mean returns, t-tests, and distribution of errors to diagnose look-ahead bias vs. distraction effects.

(a)

Experimental results

Research questions

• RQ1 Does anonymizing company identifiers reduce look-ahead bias in GPT-based sentiment trading?
• RQ2 Is distraction from general knowledge about companies more impactful than look-ahead bias in in-sample backtests?
• RQ3 How does company size influence the bias and performance of GPT-driven trading signals?
• RQ4 Do de-biased (anonymized) signals improve out-of-sample performance?
• RQ5 What is the predictive power of GPT sentiment scores under original vs. anonymized prompts?

Key findings

• In-sample, replaced headlines yielded higher average returns than original headlines for both scraped (mean 30.97 vs. 25.08; difference 5.89 bp/day; p=0.017) and Thomson Reuters data (mean 13.84 vs. 10.74; difference 4.37 bp/day; p=0.017).
• Out-of-sample, scraped data showed higher means for original vs. replaced (16.32 vs. 11.09 bp/day; not statistically significant at conventional levels), while Thomson Reuters data showed 12.23 vs. 6.07 bp/day (p≈0.064).
• The surprising in-sample result implies distraction effects outweigh look-ahead bias, especially for larger companies, where anonymization has a stronger effect.
• Classification analysis indicates original responses are more often wrong when replaced responses are neutral, contributing to larger in-sample losses under original prompts.
• Out-of-sample results suggest look-ahead bias is not the dominant concern, but distraction remains possible, with some evidence of improved performance under anonymization for large-cap stocks.
• The anonymization approach is proposed as a practical tool for de-biasing backtests and potentially improving out-of-sample performance.

(b)