[Paper Review] FinTral: A Family of GPT-4 Level Multimodal Financial Large Language Models

Gagan Bhatia, El Moatez Billah Nagoudi+2 more|arXiv (Cornell University)|Feb 16, 2024

Stock Market Forecasting MethodsDecision Sciences3 citations

TL;DR

FinTral introduces a family of multimodal financial LLMs based on Mistral-7b, enhanced with domain-specific pretraining, instruction tuning, and Direct Preference Optimization (DPO) using GPT-4-generated feedback. Its best variant, FinTral-DPO-T&R, achieves GPT-4-level performance on five of nine financial tasks and outperforms both ChatGPT-3.5 and GPT-4 in zero-shot evaluation across all tasks, demonstrating state-of-the-art multimodal financial reasoning capabilities.

ABSTRACT

We introduce FinTral, a suite of state-of-the-art multimodal large language models (LLMs) built upon the Mistral-7b model and tailored for financial analysis. FinTral integrates textual, numerical, tabular, and image data. We enhance FinTral with domain-specific pretraining, instruction fine-tuning, and RLAIF training by exploiting a large collection of textual and visual datasets we curate for this work. We also introduce an extensive benchmark featuring nine tasks and 25 datasets for evaluation, including hallucinations in the financial domain. Our FinTral model trained with direct preference optimization employing advanced Tools and Retrieval methods, dubbed FinTral-DPO-T&R, demonstrates an exceptional zero-shot performance. It outperforms ChatGPT-3.5 in all tasks and surpasses GPT-4 in five out of nine tasks, marking a significant advancement in AI-driven financial technology. We also demonstrate that FinTral has the potential to excel in real-time analysis and decision-making in diverse financial contexts. The GitHub repository for FinTral is available at \url{https://github.com/UBC-NLP/fintral}.

Motivation & Objective

• To address the limitations of existing LLMs in financial analysis, particularly their struggle with numerical reasoning, domain-specific jargon, and hallucinations in financial contexts.
• To develop a multimodal LLM capable of processing textual, numerical, tabular, and visual financial data for comprehensive document understanding.
• To create a large-scale, comprehensive benchmark—FinSet—for evaluating financial LLMs, including hallucination detection across 25 datasets and nine tasks.
• To demonstrate that a smaller, domain-adapted LLM can surpass larger models like GPT-4 in key financial reasoning tasks through advanced alignment and retrieval-augmented prompting.

Proposed method

• FinTral is built on the Mistral-7b architecture and fine-tuned via domain-specific pretraining on curated financial text and visual data.
• Instruction tuning is applied using a large-scale, curated instruction dataset tailored for financial tasks such as sentiment analysis, entity recognition, and financial reasoning.
• Direct Preference Optimization (DPO) is used to align the model with GPT-4-generated responses, improving alignment without reinforcement learning from human feedback (RLHF).
• The model is enhanced with retrieval and tool-augmented prompting (T&R), enabling dynamic access to external knowledge during inference.
• Vision capabilities are added via CLIP vision encoder, creating FinTralVL for multimodal understanding of financial documents with images or charts.
• An extensive benchmark, FinSet, is constructed with 25 datasets across nine tasks, including hallucination detection, to rigorously evaluate model performance.

Figure 1: Comparative Performance Analysis on text-based tasks of Key Financial AI Models. We compare three variations of FinTral with ChatGPT (GPT-3.5) and GPT-4 across seven task clusters: Sentiment Analysis (SA), Named Entity Recognition (NER), Number Understanding (NU), Text Summarization (TS),

Experimental results

Research questions

• RQ1 Can a smaller, domain-adapted LLM achieve GPT-4-level performance in financial reasoning tasks?
• RQ2 To what extent can DPO-based alignment improve zero-shot performance in financial LLMs compared to traditional RLHF?
• RQ3 How effective is retrieval and tool-augmentation in enhancing financial reasoning capabilities of LLMs?
• RQ4 Can a multimodal LLM effectively process and reason over textual, numerical, tabular, and visual financial data simultaneously?
• RQ5 How well can a financial LLM detect and avoid hallucinations in complex financial contexts?

Key findings

• FinTral-DPO-T&R outperforms ChatGPT-3.5 in all nine tasks evaluated on the FinSet benchmark, demonstrating superior zero-shot generalization.
• The model surpasses GPT-4 in five out of nine text-based financial tasks, including sentiment analysis, named entity recognition, and number understanding.
• FinTral-DPO achieves state-of-the-art performance on all evaluated financial tasks despite being significantly smaller than GPT-4.
• The inclusion of retrieval and tool-augmented prompting (T&R) significantly enhances reasoning and factual consistency, particularly in complex financial reasoning tasks.
• FinSet, the benchmark used, is the largest and most comprehensive financial evaluation suite to date, uniquely measuring hallucinations across 25 diverse datasets.
• The model exhibits strong multimodal capabilities, effectively integrating textual, numerical, tabular, and visual inputs for holistic financial document understanding.

Figure 2 : FinSET, a Financial Training and Evaluation Benchmark.