[Paper Review] DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuning

Wei Chen, Qiushi Wang+9 more|arXiv (Cornell University)|Oct 23, 2023

Topic Modeling4 citations

TL;DR

This paper proposes DISC-FinLLM, a Chinese financial large language model enhanced via a Multiple Experts Fine-tuning Framework (MEFF) using LoRA adaptation on four task-specific instruction datasets: financial consulting, NLP tasks, computation, and retrieval-augmented generation. The model outperforms base LLMs and existing financial models across multiple benchmarks, demonstrating superior performance in financial question answering, reasoning, and knowledge retrieval, especially when augmented with retrieval and computational plugins.

ABSTRACT

We propose Multiple Experts Fine-tuning Framework to build a financial large language model (LLM), DISC-FinLLM. Our methodology improves general LLMs by endowing them with multi-turn question answering abilities, domain text processing capabilities, mathematical computation skills, and retrieval-enhanced generation capabilities. We build a financial instruction-tuning dataset named DISC-FIN-SFT, including instruction samples of four categories (consulting, NLP tasks, computing and retrieval-augmented generation). Evaluations conducted on multiple benchmarks demonstrate that our model performs better than baseline models in various financial scenarios. Further resources can be found at https://github.com/FudanDISC/DISC-FinLLM.

Motivation & Objective

• To address the limitations of general-purpose LLMs in handling complex financial tasks in the Chinese language context.
• To overcome the scarcity of labeled financial data and lack of domain-specific knowledge in existing financial NLP models.
• To develop a unified, open-source financial LLM capable of supporting diverse financial user groups across multiple tasks.
• To enhance LLMs with multi-turn dialogue, numerical reasoning, and retrieval-augmented generation for real-world financial applications.
• To evaluate the effectiveness of task-specific LoRA fine-tuning and retrieval-enhanced generation in financial reasoning and decision support.

Proposed method

• The authors construct DISC-FIN-SFT, a supervised instruction dataset with four categories: financial consulting, NLP tasks, financial computation, and retrieval-augmented generation.
• They employ a Multiple Experts Fine-tuning Framework (MEFF) to train four separate LoRA adapters on the four data subsets, enabling task-specific specialization.
• Each LoRA adapter is fine-tuned on the Baichuan-13B-Chat base model to specialize in one of four financial capabilities: dialogue, NLP, computation, or retrieval.
• System prompts are used to guide the model’s behavior during inference, enhancing performance across diverse financial scenarios.
• The model integrates computational plugins to improve accuracy in financial formula construction and numerical reasoning tasks.
• Evaluation is conducted using multiple benchmarks and GPT-3.5-based human evaluation for retrieval and reasoning tasks, with metrics including accuracy, usefulness, linguistic quality, and reflectiveness.

Experimental results

Research questions

• RQ1 Can a multi-expert fine-tuning framework improve a general Chinese LLM’s performance across diverse financial NLP tasks?
• RQ2 How effective is task-specific LoRA fine-tuning in enhancing a base LLM’s capabilities in financial consulting, computation, and retrieval?
• RQ3 To what extent does retrieval-augmented generation improve factual accuracy and analytical depth in financial question answering?
• RQ4 How does the integration of computational plugins affect the model’s performance in financial numerical reasoning tasks?
• RQ5 Does the MEFF framework enable better generalization and adaptability than full-parameter fine-tuning on a single, unified dataset?

Key findings

• DISC-FinLLM achieves an average performance gain of 2 to 9 points over the base Baichuan-13B-Chat model on six unseen financial NLP tasks after LoRA fine-tuning.
• The model outperforms existing financial LLMs such as FinGPT-v3 and achieves the best average score among non-proprietary models in human evaluation, second only to GPT-4 and ChatGPT.
• In financial computation tasks, the model with computational plugins surpasses even ChatGPT by 0.09 points in accuracy, demonstrating the effectiveness of plugin integration.
• For retrieval-based tasks, the model achieves significantly higher scores across all four GPT-3.5 evaluation metrics—accuracy, usefulness, linguistic quality, and reflectiveness—compared to baseline models.
• Ablation studies confirm that full-parameter fine-tuning underperforms compared to task-specific LoRA fine-tuning, validating the necessity of the MEFF approach.
• The model demonstrates strong generalization, achieving high performance even on NLP tasks not covered in its training data, indicating robust transfer learning capabilities.