[Paper Review] CatMemo at the FinLLM Challenge Task: Fine-Tuning Large Language Models using Data Fusion in Financial Applications

Yupeng Cao, Zhiyuan Yao+2 more|arXiv (Cornell University)|Jul 2, 2024

Stock Market Forecasting MethodsDecision Sciences3 citations

TL;DR

This paper presents a data fusion approach for fine-tuning Llama3-8B and Mistral-7B using Parameter-Efficient Fine-Tuning (PEFT) and LoRA to enhance performance in financial NLP tasks. It achieves improved F1 (0.5634) and ROUGE-1 (0.4920) scores on classification and summarization tasks, but no improvement in single-stock trading due to task complexity and model limitations.

ABSTRACT

The integration of Large Language Models (LLMs) into financial analysis has garnered significant attention in the NLP community. This paper presents our solution to IJCAI-2024 FinLLM challenge, investigating the capabilities of LLMs within three critical areas of financial tasks: financial classification, financial text summarization, and single stock trading. We adopted Llama3-8B and Mistral-7B as base models, fine-tuning them through Parameter Efficient Fine-Tuning (PEFT) and Low-Rank Adaptation (LoRA) approaches. To enhance model performance, we combine datasets from task 1 and task 2 for data fusion. Our approach aims to tackle these diverse tasks in a comprehensive and integrated manner, showcasing LLMs' capacity to address diverse and complex financial tasks with improved accuracy and decision-making capabilities.

Motivation & Objective

• To improve financial NLP performance by fusing datasets from multiple tasks using fine-tuned LLMs.
• To evaluate the effectiveness of PEFT and LoRA in adapting LLMs for financial classification, summarization, and trading.
• To investigate whether combining data from classification and summarization tasks enhances reasoning and generalization in financial LLMs.
• To assess the transferability of fine-tuned models to complex, sequential decision-making tasks like single-stock trading.
• To identify limitations of smaller LLMs (7B/8B) in handling high-stakes financial prediction tasks.

Proposed method

• Fine-tuned Llama3-8B and Mistral-7B using LoRA and PEFT for parameter-efficient adaptation.
• Fused training data from Task 1 (classification) and Task 2 (summarization) to enhance model generalization.
• Used ROUGE-1, ROUGE-2, BERTScore, and F1 as evaluation metrics for summarization and classification tasks.
• Evaluated trading performance using Sharpe Ratio, Cumulative Return, and Volatility on four stocks.
• Implemented models in the FinMem framework to generate sequential trading decisions from financial text.
• Selected Mistral-7B fine-tuned on fused data for final testing due to consistent output formatting and performance.

Experimental results

Research questions

• RQ1 Does data fusion between financial classification and summarization tasks improve LLM performance on downstream NLP tasks?
• RQ2 Can PEFT and LoRA effectively adapt 7B/8B LLMs to financial text understanding with limited labeled data?
• RQ3 Does fine-tuning on combined datasets enhance reasoning and generalization across diverse financial tasks?
• RQ4 Why do fine-tuned models fail to improve performance in the more complex single-stock trading task?
• RQ5 How do model size and architecture (e.g., Mistral-7B vs. Llama3-8B) affect performance in financial decision-making tasks?

Key findings

• Mistral-7B outperformed Llama3-8B in both classification and summarization tasks, showing better output structure and performance.
• The fused-data fine-tuned model achieved an F1 score of 0.5634 on Task 1, a significant improvement over single-task fine-tuning.
• For summarization, the fused model achieved a ROUGE-1 score of 0.4920, showing enhanced summarization quality.
• The model fine-tuned on fused data did not improve performance on the single-stock trading task (Task 3), with a Sharpe Ratio of -0.6199 on the test set.
• Model 3, trained on both Task 1 and Task 2 data, did not outperform models trained on individual tasks, suggesting potential noise or conflicting signals from fused data.
• All models performed poorly in trading due to the complexity of sequential decision-making and limitations of 7B/8B LLMs in handling high-stakes financial prediction.