[Paper Review] PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance

Qianqian Xie, Weiguang Han+5 more|arXiv (Cornell University)|Jun 8, 2023

Stock Market Forecasting Methods43 citations

TL;DR

PIXIU presents FinMA, an open-source financial LLM fine-tuned from LLaMA with a large multi-task, multi-modal instruction dataset (FIT) and an evaluation benchmark (FLARE); FinMA excels on financial NLP tasks but shows room for improvement in complex QA and stock-movement prediction.

ABSTRACT

Although large language models (LLMs) has shown great performance on natural language processing (NLP) in the financial domain, there are no publicly available financial tailtored LLMs, instruction tuning datasets, and evaluation benchmarks, which is critical for continually pushing forward the open-source development of financial artificial intelligence (AI). This paper introduces PIXIU, a comprehensive framework including the first financial LLM based on fine-tuning LLaMA with instruction data, the first instruction data with 136K data samples to support the fine-tuning, and an evaluation benchmark with 5 tasks and 9 datasets. We first construct the large-scale multi-task instruction data considering a variety of financial tasks, financial document types, and financial data modalities. We then propose a financial LLM called FinMA by fine-tuning LLaMA with the constructed dataset to be able to follow instructions for various financial tasks. To support the evaluation of financial LLMs, we propose a standardized benchmark that covers a set of critical financial tasks, including five financial NLP tasks and one financial prediction task. With this benchmark, we conduct a detailed analysis of FinMA and several existing LLMs, uncovering their strengths and weaknesses in handling critical financial tasks. The model, datasets, benchmark, and experimental results are open-sourced to facilitate future research in financial AI.

Motivation & Objective

• Motivate the need for open, instruction-following financial LLMs and curated datasets.
• Create FinMA by fine-tuning LLaMA with multi-task, multi-modal financial instructions.
• Develop FIT, the first large-scale financial instruction tuning dataset (136K samples).
• Propose FLARE, a holistic benchmark including financial NLP and prediction tasks.
• Provide open resources to advance research in financial AI.

Proposed method

• Construct FIT from open-source financial datasets spanning NLP and stock prediction, with multi-modal data (text, tables, time-series).
• Design domain-specific instructions per task and assemble instruction-tuning samples (Instruction, Text/Context, Response).
• Fine-tune LLaMA models (7B and 30B, plus a 7B-full variant) on FIT using AdamW, specified hyperparameters, and multi-epoch schedules.
• Create FLARE with 4 financial NLP tasks across 6 datasets and 1 financial prediction task across 3 datasets; standard evaluation metrics per task.
• Compare FinMA against BloombergGPT, GPT-4, ChatGPT, BLOOM, GPT-NeoX, OPT-66B, and Vicuna-13B on FLARE in zero-shot and few-shot settings.

Experimental results

Research questions

• RQ1 Can open financial instruction data and models close the gap with proprietary LLMs in finance?
• RQ2 How does multi-task, multi-modal instruction tuning affect FinMA’s performance on financial tasks?
• RQ3 What are the strengths and limitations of FinMA compared with baselines on financial NLP and prediction tasks?
• RQ4 How do model size and instruction data quality impact performance across FLARE tasks?

Key findings

• FinMA significantly outperforms several LLMs on FPB, FiQA-SA, and Headline NLP tasks (e.g., FinMA-30B surpasses GPT-4 on FPB by about 10% F1; BloombergGPT by ~37% F1).
• FinMA also achieves competitive NER results, outperforming BloombergGPT and others on the NER task.
• On complex numeric reasoning tasks (FinQA, ConvFinQA), FinMA lags behind GPT-4 and BloombergGPT due to limitations in quantitative reasoning of the backbone model (LLaMA).
• For stock movement prediction, all LLMs show limited performance; FinMA-7B-full shows improvements on ACL18 but remains weak on other datasets, highlighting task difficulty.
• FinMA-full (trained with NLP and prediction data) demonstrates strongest performance on ACL18 and competitive NLP results, indicating value of domain-aligned, task-comprehensive fine-tuning.
• The study highlights that instruction data quality and task alignment can be more critical than merely increasing model size for many tasks.