[Paper Review] Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications

Jimin Huang, Mengxi Xiao+42 more|arXiv (Cornell University)|Aug 20, 2024

Stock Market Forecasting Methods4 citations

TL;DR

Open-FinLLMs introduces a suite of open-source, multimodal large language models—FinLLaMA, FinLLaMA-instruct, and FinLLaVA—designed for financial applications by pre-training on a 52B-token financial corpus including text, tables, and time-series data, followed by instruction tuning and multimodal alignment. The models achieve state-of-the-art performance on 19 zero-shot and 4 few-shot financial benchmarks, outperforming LLaMA3-8B, BloombergGPT, and even GPT-4 in several tasks, with FinLLaMA delivering high Sharpe Ratios in trading simulations.

ABSTRACT

Financial LLMs hold promise for advancing financial tasks and domain-specific applications. However, they are limited by scarce corpora, weak multimodal capabilities, and narrow evaluations, making them less suited for real-world application. To address this, we introduce extit{Open-FinLLMs}, the first open-source multimodal financial LLMs designed to handle diverse tasks across text, tabular, time-series, and chart data, excelling in zero-shot, few-shot, and fine-tuning settings. The suite includes FinLLaMA, pre-trained on a comprehensive 52-billion-token corpus; FinLLaMA-Instruct, fine-tuned with 573K financial instructions; and FinLLaVA, enhanced with 1.43M multimodal tuning pairs for strong cross-modal reasoning. We comprehensively evaluate Open-FinLLMs across 14 financial tasks, 30 datasets, and 4 multimodal tasks in zero-shot, few-shot, and supervised fine-tuning settings, introducing two new multimodal evaluation datasets. Our results show that Open-FinLLMs outperforms afvanced financial and general LLMs such as GPT-4, across financial NLP, decision-making, and multi-modal tasks, highlighting their potential to tackle real-world challenges. To foster innovation and collaboration across academia and industry, we release all codes (https://anonymous.4open.science/r/PIXIU2-0D70/B1D7/LICENSE) and models under OSI-approved licenses.

Motivation & Objective

• Address the lack of comprehensive financial knowledge and multimodal capabilities in general-purpose LLMs for financial applications.
• Overcome limitations in existing financial LLMs that underutilize tabular and time-series data during pre-training and fine-tuning.
• Improve generalization and real-world applicability by evaluating models across diverse financial scenarios, including trading simulations.
• Provide open-source, instruction-tuned, and multimodal financial LLMs to accelerate research and innovation in finance and AI.

Proposed method

• Pre-train FinLLaMA on a 52 billion token financial corpus containing text, tables, and time-series data to embed deep financial knowledge.
• Fine-tune FinLLaMA with 573K financial instruction pairs to create FinLLaMA-instruct, enhancing zero-shot and few-shot performance on financial tasks.
• Train FinLLaVA on 1.43 million image-text instruction pairs to enable multimodal understanding of financial charts, tables, and reports.
• Integrate the models into a memory-augmented agent (FinMem) that retrieves and reasons over short-, mid-, long-, and reflection-term insights for trading decisions.
• Evaluate models across 19 zero-shot and 4 few-shot datasets, including financial reasoning, entity recognition, and sentiment analysis.
• Conduct trading simulations using cumulative return and Sharpe ratio metrics to assess real-world performance on stocks like TSLA, COIN, GOOG, and NIO.

Experimental results

Research questions

• RQ1 Can a large language model pre-trained on a diverse financial corpus including tables and time-series data outperform general-purpose LLMs in zero-shot and few-shot financial reasoning tasks?
• RQ2 To what extent does instruction fine-tuning on 573K financial instructions improve performance on downstream financial NLP tasks compared to base models?
• RQ3 How well can a multimodal LLM (FinLLaVA) understand and reason about financial charts and tables compared to unimodal models?
• RQ4 Can the integration of a memory-augmented reasoning agent (FinMem) with Open-FinLLMs lead to superior trading performance in simulated environments?
• RQ5 What is the real-world financial performance of Open-FinLLMs in terms of cumulative returns and Sharpe ratios across multiple stocks and time periods?

Key findings

• FinLLaMA outperforms LLaMA3-8B, LLaMA3.1-8B, and BloombergGPT across 19 zero-shot and 4 few-shot financial benchmark datasets.
• FinLLaMA-instruct surpasses GPT-4 and other financial LLMs on 15 out of 15 evaluated financial reasoning and classification tasks.
• FinLLaVA demonstrates superior performance in understanding tables and charts across 4 multimodal financial tasks, indicating strong visual-linguistic alignment.
• In trading simulations, FinLLaMA achieves high cumulative returns and strong Sharpe Ratios on TSLA, COIN, and NIO, outperforming baseline models and the Buy & Hold strategy.
• The FinMem agent, powered by Open-FinLLMs, generates consistent and well-justified trading decisions by integrating insights from multiple memory layers with reasoning and reflection.
• The models show robust generalization across diverse financial scenarios, including momentum-based trading, earnings expectations, and market sentiment analysis.