[Paper Review] Financial Statement Analysis with Large Language Models

Alex Kim, Maximilian Muhn+1 more|arXiv (Cornell University)|Jul 25, 2024

Stock Market Forecasting Methods12 citations

TL;DR

The paper shows that an LLM (GPT-4) can analyze anonymized financial statements and predict the direction of future earnings, outperforming human analysts and matching or surpassing narrow ML models, with narrative insights driving its performance.

ABSTRACT

We investigate whether large language models (LLMs) can successfully perform financial statement analysis in a way similar to a professional human analyst. We provide standardized and anonymous financial statements to GPT4 and instruct the model to analyze them to determine the direction of firms' future earnings. Even without narrative or industry-specific information, the LLM outperforms financial analysts in its ability to predict earnings changes directionally. The LLM exhibits a relative advantage over human analysts in situations when the analysts tend to struggle. Furthermore, we find that the prediction accuracy of the LLM is on par with a narrowly trained state-of-the-art ML model. LLM prediction does not stem from its training memory. Instead, we find that the LLM generates useful narrative insights about a company's future performance. Lastly, our trading strategies based on GPT's predictions yield a higher Sharpe ratio and alphas than strategies based on other models. Our results suggest that LLMs may take a central role in analysis and decision-making.

Motivation & Objective

• Assess whether a large language model can perform financial statement analysis like professional analysts using only numeric data.
• Compare LLM performance to human analysts and to narrow ML models (logistic regression and ANN) in predicting earnings direction.
• Investigate whether chain-of-thought prompting enhances LLM performance and whether LLM narratives add value.
• Examine the economic usefulness of LLM forecasts for asset pricing and trading performance.

Proposed method

• Provide standardized and anonymized balance sheets and income statements to GPT-4 Turbo, with no accompanying narrative text.
• Use two prompting regimes: a simple prompt and a Chain-of-Thought (CoT) prompt that mimics analyst reasoning.
• Evaluate predictions against analyst consensus forecasts from IBES and against five-year rolling out-of-sample forecasts.
• Compare GPT-4 CoT performance to logistic regression and an artificial neural network trained on the same financial statement data.
• Analyze whether GPT’s predictive narratives contain information about future earnings by using embeddings to train an auxiliary predictor.

Experimental results

Research questions

• RQ1 Can an LLM generate economic insights and predict the direction of earnings using only numeric financial statement data without narrative context?
• RQ2 How does LLM performance compare to human analysts and to narrow ML models in predicting earnings direction?
• RQ3 Does chain-of-thought prompting and narrative generation enhance LLM forecasting, and what is the incremental value of LLM narratives?
• RQ4 Are LLM forecasts economically useful for asset pricing and trading strategies relative to benchmarks?
• RQ5 Do LLM predictions generalize out-of-sample beyond the model’s training window?

Key findings

• GPT-4 with chain-of-thought prompts achieves higher accuracy (60%) in predicting the direction of next-year earnings than the average analyst benchmarks in the study.
• GPT-4’s accuracy is on par with or slightly higher than a state-of-the-art ANN trained on the same data, with a higher F1-score for GPT.
• Narratives generated by the CoT prompts contain information that, when used in an embedding-based model, predicts earnings direction with competitive accuracy, suggesting the insights drive performance.
• GPT’s forecasts yield trading relevance, with long-short strategies based on GPT forecasts delivering significant alphas and Sharpe ratios, especially for small-cap firms.
• The model’s effectiveness is not due to memory of training data; out-of-sample tests on 2023 data confirm performance beyond the model’s training window.