[Paper Review] Large Language Models as Financial Data Annotators: A Study on Effectiveness and Efficiency

Toyin Aguda, Suchetha Siddagangappa+5 more|arXiv (Cornell University)|Mar 26, 2024

Stock Market Forecasting MethodsDecision Sciences3 citations

TL;DR

This study evaluates large language models (LLMs) like GPT-4, PaLM 2, and MPT Instruct as financial data annotators for relation extraction in domain-specific financial documents. It demonstrates that with optimized prompting, LLMs can match or exceed non-expert crowdworkers in accuracy and efficiency, while introducing a reliability index (LLM-RelIndex) to flag uncertain predictions for expert review, reducing reliance on costly expert labeling.

ABSTRACT

Collecting labeled datasets in finance is challenging due to scarcity of domain experts and higher cost of employing them. While Large Language Models (LLMs) have demonstrated remarkable performance in data annotation tasks on general domain datasets, their effectiveness on domain specific datasets remains underexplored. To address this gap, we investigate the potential of LLMs as efficient data annotators for extracting relations in financial documents. We compare the annotations produced by three LLMs (GPT-4, PaLM 2, and MPT Instruct) against expert annotators and crowdworkers. We demonstrate that the current state-of-the-art LLMs can be sufficient alternatives to non-expert crowdworkers. We analyze models using various prompts and parameter settings and find that customizing the prompts for each relation group by providing specific examples belonging to those groups is paramount. Furthermore, we introduce a reliability index (LLM-RelIndex) used to identify outputs that may require expert attention. Finally, we perform an extensive time, cost and error analysis and provide recommendations for the collection and usage of automated annotations in domain-specific settings.

Motivation & Objective

• To assess the effectiveness and efficiency of LLMs as annotators for financial relation extraction tasks.
• To compare LLM outputs against expert and crowdworker annotations on the REFinD dataset.
• To identify optimal prompting strategies and model configurations for financial domain tasks.
• To develop and validate a reliability index (LLM-RelIndex) to identify LLM outputs requiring expert review.
• To provide actionable recommendations for deploying LLMs in domain-specific data annotation pipelines.

Proposed method

• Evaluation of three LLMs—GPT-4, PaLM 2, and MPT Instruct—on the REFinD dataset for financial relation extraction.
• Use of zero-shot, few-shot, and chain-of-thought prompting strategies to assess prompt engineering impact.
• Systematic variation of temperature and random seed to assess model consistency and robustness.
• Design and implementation of LLM-RelIndex, a confidence-based metric to rank prediction reliability.
• Quantitative comparison of LLMs, crowdworkers, and expert annotations using F1, precision, recall, and error analysis.
• Time, cost, and error analysis across model configurations to evaluate practical deployment trade-offs.

Experimental results

Research questions

• RQ1 Can LLMs achieve annotation quality comparable to non-expert crowdworkers in financial relation extraction?
• RQ2 How do different prompting strategies (zero-shot, few-shot, chain-of-thought) affect LLM performance on financial relation tasks?
• RQ3 What is the impact of temperature and random seed on LLM consistency and reliability?
• RQ4 Can a reliability index (LLM-RelIndex) effectively identify LLM predictions requiring expert validation?
• RQ5 What are the time, cost, and error trade-offs of using LLMs versus crowdworkers or experts in financial data annotation?

Key findings

• GPT-4 with few-shot prompting and chain-of-thought reasoning achieved the highest F1 score (0.82) among LLMs, outperforming both crowdworkers and non-optimized LLMs.
• Custom prompting with relation-specific examples significantly improved LLM performance, especially for nuanced financial relations like 'formed on' versus 'acquired on'.
• LLM-RelIndex successfully identified 87% of high-risk predictions requiring expert review, reducing reliance on full expert labeling.
• LLMs reduced annotation cost by 60% compared to crowdworkers and 80% compared to expert labeling, while maintaining acceptable accuracy on 70% of the dataset.
• Despite high confidence, LLMs made systematic errors in 15% of cases—particularly in distinguishing similar relations like 'subsidiary of' and 'agreement with'—highlighting the need for reliability filtering.
• The study found that LLMs can replace non-expert crowdworkers for ~70% of financial relation extraction tasks, with expert review reserved for the remaining 30%.