[Paper Review] Empowering Many, Biasing a Few: Generalist Credit Scoring through Large Language Models

Duanyu Feng, Y. S. Dai+7 more|arXiv (Cornell University)|Oct 1, 2023

Financial Distress and Bankruptcy Prediction12 citations

TL;DR

The paper introduces CALM, an instruction-tuned LLM framework for generalist credit scoring across multiple online financial tasks, with a benchmark of 9 datasets and an emphasis on bias analysis and open resources.

ABSTRACT

In the financial industry, credit scoring is a fundamental element, shaping access to credit and determining the terms of loans for individuals and businesses alike. Traditional credit scoring methods, however, often grapple with challenges such as narrow knowledge scope and isolated evaluation of credit tasks. Our work posits that Large Language Models (LLMs) have great potential for credit scoring tasks, with strong generalization ability across multiple tasks. To systematically explore LLMs for credit scoring, we propose the first open-source comprehensive framework. We curate a novel benchmark covering 9 datasets with 14K samples, tailored for credit assessment and a critical examination of potential biases within LLMs, and the novel instruction tuning data with over 45k samples. We then propose the first Credit and Risk Assessment Large Language Model (CALM) by instruction tuning, tailored to the nuanced demands of various financial risk assessment tasks. We evaluate CALM, existing state-of-art (SOTA) methods, open source and closed source LLMs on the build benchmark. Our empirical results illuminate the capability of LLMs to not only match but surpass conventional models, pointing towards a future where credit scoring can be more inclusive, comprehensive, and unbiased. We contribute to the industry's transformation by sharing our pioneering instruction-tuning datasets, credit and risk assessment LLM, and benchmarks with the research community and the financial industry.

Motivation & Objective

• Demonstrate that LLMs can generalize across diverse online credit and risk tasks beyond single-task expert systems.
• Create and release a comprehensive benchmark of 9 datasets (~14K samples) for credit and risk assessment.
• Develop CALM, an instruction-tuned LLM tailored to credit and risk tasks using a large instruction-tuning corpus.
• Investigate potential biases in LLMs when applied to credit scoring and risk assessment and propose ethical considerations.

Proposed method

• Construct a diverse, tabular-data benchmark spanning credit scoring, fraud detection, financial distress, and claim analysis with 14K samples.
• Assemble a 45K-instruction-tuning dataset (6 datasets, balanced via resampling) and use prompts in table-based and description-based forms.
• Fine-tune an LLaMa2-chat model using LoRA with 5 epochs, AdamW, learning rate 3e-4, weight decay 1e-5, max input length 2048.
• Evaluate CALM against SOTA expert systems and multiple open/non-open LLMs (e.g., GPT-4, ChatGPT, Bloomz, Vicuna, Llama variants) on accuracy, F1, MCC, and bias metrics.
• Analyze data bias (Disparate Impact) and model bias (Equal Opportunity Difference, Average Odds Difference) following AI FAIRNESS 360.

Experimental results

Research questions

• RQ1 H1: Can LLMs overcome narrow expertise in traditional credit/risk systems by leveraging broad pretraining for diverse online tasks?
• RQ2 H2: Can instruction-tuned LLMs generalize/accommodate multiple related credit tasks through fine-tuning on financial data?
• RQ3 H3: Do advances in LLM capabilities introduce or amplify fairness biases in credit decisions?

Key findings

• LLMs, particularly GPT-4, can match or surpass some conventional models on several credit/risk tasks.
• CALM, the fine-tuned LLM, demonstrates transfer of knowledge across multiple credit/risk tasks and improves performance on untrained datasets.
• There remain observable biases in LLMs on sensitive attributes, underscoring the need for ethical oversight in deployment.