[Paper Review] A Comparative Analysis of Fine-Tuned LLMs and Few-Shot Learning of LLMs for Financial Sentiment Analysis

Sorouralsadat Fatemi, Yuheng Hu|arXiv (Cornell University)|Dec 14, 2023

Stock Market Forecasting Methods8 citations

TL;DR

The paper compares fine-tuned smaller LLMs (250M–3B params) with zero-/few-shot in-context learning of LLMs for financial sentiment analysis, finding fine-tuned models can match state-of-the-art performance with less compute, while zero-/one-shot results vary across datasets and models.

ABSTRACT

Financial sentiment analysis plays a crucial role in uncovering latent patterns and detecting emerging trends, enabling individuals to make well-informed decisions that may yield substantial advantages within the constantly changing realm of finance. Recently, Large Language Models (LLMs) have demonstrated their effectiveness in diverse domains, showcasing remarkable capabilities even in zero-shot and few-shot in-context learning for various Natural Language Processing (NLP) tasks. Nevertheless, their potential and applicability in the context of financial sentiment analysis have not been thoroughly explored yet. To bridge this gap, we employ two approaches: in-context learning (with a focus on gpt-3.5-turbo model) and fine-tuning LLMs on a finance-domain dataset. Given the computational costs associated with fine-tuning LLMs with large parameter sizes, our focus lies on smaller LLMs, spanning from 250M to 3B parameters for fine-tuning. We then compare the performances with state-of-the-art results to evaluate their effectiveness in the finance-domain. Our results demonstrate that fine-tuned smaller LLMs can achieve comparable performance to state-of-the-art fine-tuned LLMs, even with models having fewer parameters and a smaller training dataset. Additionally, the zero-shot and one-shot performance of LLMs produces comparable results with fine-tuned smaller LLMs and state-of-the-art outcomes. Furthermore, our analysis demonstrates that there is no observed enhancement in performance for finance-domain sentiment analysis when the number of shots for in-context learning is increased.

Motivation & Objective

• Investigate the effectiveness of zero-shot and few-shot learning for financial sentiment analysis using LLMs.
• Evaluate fine-tuning of smaller LLMs (Flan-T5 variants) on finance-domain data.
• Compare results with state-of-the-art models (FinBERT, Instruct-FinGPT) to assess trade-offs in accuracy and computational cost.

Proposed method

• Study zero-shot and few-shot settings with Flan-T5 Base/Large/XL and gpt-3.5-turbo (ChatGPT).
• Fine-tune Flan-T5 models using QLoRA with 4-bit quantization on finance data (Twitter Financial News Sentiment) across 3 epochs.
• Compare zero-/few-shot results to FinBert and Instruct-FinGPT on FPB and TFSN datasets.
• Use prompts with and without label descriptions to assess prompt design impact.
• Report accuracy and F1-macro for sentiment labels (Positive, Negative, Neutral).
• Evaluate training efficiency and memory usage on single A100 GPUs.

Experimental results

Research questions

• RQ1 How do zero-shot and one-/few-shot LLM in-context learning performances compare to fine-tuned small LLMs for financial sentiment analysis?
• RQ2 Can fine-tuning small LLMs (250M–3B parameters) achieve comparable results to state-of-the-art models with reduced computational resources?
• RQ3 How does model size and prompt design affect zero-/few-shot performance on finance-specific sentiment tasks?
• RQ4 Do FPB and TFSN datasets show consistent trends in zero-/few-shot vs fine-tuned performance?

Key findings

• Fine-tuned Flan-T5 models achieve performance comparable to state-of-the-art fine-tuned LLMs using far fewer computational resources.
• Zero-shot performance of ChatGPT reaches about 82% on TFSN but remains below in-context fine-tuned models, while FPB shows zero-shot performance closer to fine-tuned Flan-T5 models.
• Zero-shot results on FPB improve with larger models, while FPB shows less sensitivity to prompt structure than TFSN.
• One-shot prompts generally improve performance for most models, but five- and ten-shot settings yield inconsistent gains across datasets and models.
• Flan-T5 fine-tuning with QLoRA on finance data provides faster convergence and lower memory usage (trained on a single A100) compared to more extensive multi-GPU setups.
• The FPB dataset favors zero-/one-shot performance for some smaller models, while TFSN tends to favor fine-tuned approaches, highlighting dataset-specific dynamics in sentiment tasks.