[Paper Review] Chinese Fine-Grained Financial Sentiment Analysis with Large Language Models

Yinyu Lan, Yanru Wu+3 more|arXiv (Cornell University)|Jun 25, 2023

Stock Market Forecasting Methods4 citations

TL;DR

This paper introduces FinChina SA, a large-scale Chinese fine-grained financial sentiment analysis dataset with 11,036 news articles, 8,739 companies, 190 warning types, and 21,272 sentiment annotations. It evaluates open-source LLMs using instruction-based and conversation-based fine-tuning, showing that Chinese LLaMA Plus achieves 75.99% accuracy in sentiment classification—surpassing models like BLOOMZ and ChatGLM—while ChatGPT underperforms in zero-shot settings due to limited financial domain knowledge.

ABSTRACT

Entity-level fine-grained sentiment analysis in the financial domain is a crucial subtask of sentiment analysis and currently faces numerous challenges. The primary challenge stems from the lack of high-quality and large-scale annotated corpora specifically designed for financial text sentiment analysis, which in turn limits the availability of data necessary for developing effective text processing techniques. Recent advancements in large language models (LLMs) have yielded remarkable performance in natural language processing tasks, primarily centered around language pattern matching. In this paper, we propose a novel and extensive Chinese fine-grained financial sentiment analysis dataset, FinChina SA, for enterprise early warning. We thoroughly evaluate and experiment with well-known existing open-source LLMs using our dataset. We firmly believe that our dataset will serve as a valuable resource to advance the exploration of real-world financial sentiment analysis tasks, which should be the focus of future research. The FinChina SA dataset is publicly available at https://github.com/YerayL/FinChina-SA

Motivation & Objective

• To address the lack of high-quality, large-scale annotated corpora for Chinese financial sentiment analysis.
• To develop a benchmark dataset for entity-level, fine-grained sentiment analysis in the financial domain.
• To evaluate the effectiveness of open-source LLMs and ChatGPT in zero-shot and fine-tuned settings for financial sentiment tasks.
• To investigate the feasibility and performance of instruction-based and conversation-based fine-tuning for financial NLP.
• To identify limitations of general-purpose LLMs like ChatGPT in domain-specific financial sentiment understanding.

Proposed method

• Crawled and cleaned financial news from major Chinese financial websites to construct the FinChina SA dataset.
• Annotated each article with company names, sentiment polarities, and warning types, resulting in 21,272 sentiment examples.
• Fine-tuned open-source LLMs using instruction-based tuning with a single-turn QA format and conversation-based tuning with multi-turn dialogue templates.
• Evaluated models using zero-shot prompting with ChatGPT and compared performance across sentiment classification and warning type detection tasks.
• Applied efficient parameter tuning methods like P-tuning v2 to reduce memory usage, though with a trade-off in accuracy.
• Used a turn-based prompting strategy in multi-turn QA fine-tuning, appending model responses to previous inputs for subsequent rounds.

Experimental results

Research questions

• RQ1 How effective are open-source LLMs in zero-shot and fine-tuned settings for Chinese fine-grained financial sentiment analysis?
• RQ2 Can instruction-based and conversation-based fine-tuning improve performance on financial sentiment classification tasks?
• RQ3 Why does ChatGPT underperform in financial sentiment analysis despite strong general NLP capabilities?
• RQ4 To what extent does domain-specific pre-training enhance LLM performance on financial text understanding?
• RQ5 Does transforming single-turn data into multi-turn dialogue formats improve model performance in financial sentiment tasks?

Key findings

• Chinese LLaMA Plus achieved the highest sentiment classification accuracy at 75.99%, outperforming BLOOMZ (75.52) and ChatGLM (74.46).
• ChatGPT performed poorly in the zero-shot setting, indicating limited understanding of financial domain-specific expressions and sentiment knowledge.
• Fine-tuned LLMs significantly outperformed the Longformer model, demonstrating effective transfer learning to financial NLP tasks.
• Conversation-based fine-tuning using template-converted multi-turn data led to repetitive and erroneous outputs, suggesting it is less effective than instruction-based tuning for this task.
• P-tuning v2 reduced memory usage but caused a notable drop in model accuracy, indicating a trade-off between efficiency and performance.
• The FinChina SA dataset is publicly available at https://github.com/YerayL/FinChina-SA, serving as a valuable benchmark for future research in financial sentiment analysis.