[Paper Review] FinPT: Financial Risk Prediction with Profile Tuning on Pretrained Foundation Models

Yuwei Yin, Yazheng Yang+2 more|arXiv (Cornell University)|Jul 22, 2023

Stock Market Forecasting Methods4 citations

TL;DR

This paper proposes FinPT, a novel method for financial risk prediction that converts tabular financial data into natural-language customer profiles using large language models (LLMs), then fine-tunes foundation models on these profiles for improved classification. The approach achieves consistent performance gains over traditional baselines across multiple financial risk tasks, with Flan-T5-Base showing the strongest results on FinBench, a new open-source benchmark for default, fraud, and churn prediction.

ABSTRACT

Financial risk prediction plays a crucial role in the financial sector. Machine learning methods have been widely applied for automatically detecting potential risks and thus saving the cost of labor. However, the development in this field is lagging behind in recent years by the following two facts: 1) the algorithms used are somewhat outdated, especially in the context of the fast advance of generative AI and large language models (LLMs); 2) the lack of a unified and open-sourced financial benchmark has impeded the related research for years. To tackle these issues, we propose FinPT and FinBench: the former is a novel approach for financial risk prediction that conduct Profile Tuning on large pretrained foundation models, and the latter is a set of high-quality datasets on financial risks such as default, fraud, and churn. In FinPT, we fill the financial tabular data into the pre-defined instruction template, obtain natural-language customer profiles by prompting LLMs, and fine-tune large foundation models with the profile text to make predictions. We demonstrate the effectiveness of the proposed FinPT by experimenting with a range of representative strong baselines on FinBench. The analytical studies further deepen the understanding of LLMs for financial risk prediction.

Motivation & Objective

• To address the stagnation in financial risk prediction methods due to outdated algorithms and lack of standardized benchmarks.
• To propose a unified, open-source benchmark (FinBench) for financial risk prediction covering default, fraud, and churn.
• To develop FinPT, a profile tuning framework that leverages LLMs to convert tabular data into natural-language profiles for improved model generalization.
• To evaluate the effectiveness of profile tuning on foundation models across diverse financial risk tasks and compare it with strong baselines.

Proposed method

• Transform tabular financial data into natural-language customer profiles by filling a pre-defined instruction template with column names and values.
• Use large language models (e.g., GPT-2, LLaMA, Flan-T5) to generate fluent, coherent profile descriptions from the templated data.
• Fine-tune large foundation models on the generated profile texts using a small feedforward classifier head for binary risk prediction.
• Train the model end-to-end with a focal loss to handle class imbalance, prioritizing correct prediction of risky (positive) instances.
• Enable joint training across all datasets in FinBench to improve generalization and model robustness.
• Explore alternative strategies such as in-context learning and instruction tuning to compare with full fine-tuning.

Experimental results

Research questions

• RQ1 Can transforming tabular financial data into natural-language profiles improve the performance of foundation models in financial risk prediction tasks?
• RQ2 How does Profile Tuning on LLMs compare to traditional tabular classification models (e.g., XGBoost, Random Forest) on financial risk prediction?
• RQ3 Does joint training on multiple financial risk datasets (default, fraud, churn) improve model generalization and performance?
• RQ4 How do different foundation models (e.g., Flan-T5, LLaMA, GPT-2) perform under the FinPT framework, and what role does instruction tuning play?
• RQ5 Can in-context learning or instruction tuning alone yield competitive results compared to full fine-tuning in this setting?

Key findings

• FinPT significantly outperforms strong baselines such as XGBoost, Random Forest, DeepFM, and TabNet across all datasets in FinBench, with consistent F1-score improvements.
• Flan-T5-Base achieves the best performance among all backbone models, outperforming even larger models like LLaMA-13B in some cases, highlighting the value of instruction tuning.
• Training on all datasets jointly via Profile Tuning leads to consistent performance gains, with the largest improvement observed on the low-performing cf2 dataset.
• LLaMA-13B achieves strong results even when only the last decoder layer is fine-tuned, demonstrating the efficiency and effectiveness of the FinPT approach.
• In-context learning and instruction tuning alone yield poor F1-scores (<10), but generate useful, explainable rationales that support human interpretation.
• GPT-2 outperforms BERT-based models by a large margin, suggesting that decoder-only architectures with last-token prediction are better suited for this classification task.