[Paper Review] Credit Risk Meets Large Language Models: Building a Risk Indicator from Loan Descriptions in P2P Lending

Mario Sanz-Guerrero, Javier Arroyo|arXiv (Cornell University)|Jan 29, 2024

FinTech, Crowdfunding, Digital FinanceBusiness, Management and Accounting3 citations

TL;DR

This paper proposes a novel credit risk scoring method for P2P lending by leveraging BERT, a large language model, to analyze borrower-provided loan descriptions. Fine-tuned via transfer learning, the model generates a risk score that significantly improves credit risk classification performance, outperforming models using only categorical variables, though interpretability and bias remain critical challenges.

ABSTRACT

Peer-to-peer (P2P) lending connects borrowers and lenders through online platforms but suffers from significant information asymmetry, as lenders often lack sufficient data to assess borrowers' creditworthiness. This paper addresses this challenge by leveraging BERT, a Large Language Model (LLM) known for its ability to capture contextual nuances in text, to generate a risk score based on borrowers' loan descriptions using a dataset from the Lending Club platform. We fine-tune BERT to distinguish between defaulted and non-defaulted loans using the loan descriptions provided by the borrowers. The resulting BERT-generated risk score is then integrated as an additional feature into an XGBoost classifier used at the loan granting stage, where decision-makers have limited information available to guide their decisions. This integration enhances predictive performance, with improvements in balanced accuracy and AUC, highlighting the value of textual features in complementing traditional inputs. Moreover, we find that the incorporation of the BERT score alters how classification models utilize traditional input variables, with these changes varying by loan purpose. These findings suggest that BERT discerns meaningful patterns in loan descriptions, encompassing borrower-specific features, specific purposes, and linguistic characteristics. However, the inherent opacity of LLMs and their potential biases underscore the need for transparent frameworks to ensure regulatory compliance and foster trust. Overall, this study demonstrates how LLM-derived insights interact with traditional features in credit risk modeling, opening new avenues to enhance the explainability and fairness of these models.

Motivation & Objective

• To address information asymmetry in P2P lending by extracting credit risk signals from unstructured loan descriptions.
• To evaluate whether large language models like BERT can improve credit risk classification beyond traditional variables.
• To develop a scalable, automated risk scoring system without manual annotation or subjective labeling.
• To assess the impact of linguistic features and content quality on the generated risk scores.
• To identify key challenges related to interpretability, bias, and regulatory compliance in LLM-driven credit assessment.

Proposed method

• Fine-tune BERT using transfer learning on the Lending Club dataset to classify loan descriptions as default or non-default.
• Use the final layer of BERT’s classification head to extract a continuous risk score from each loan description.
• Integrate the BERT-generated risk score as a feature into a XGBoost-based credit risk classifier.
• Perform extensive data preprocessing, including text cleaning, tokenization, and handling of class imbalance in the dataset.
• Apply explainability techniques to analyze how linguistic tone, clarity, and content influence the risk score.
• Compare model performance using only categorical variables versus including the BERT risk score to assess incremental value.

Experimental results

Research questions

• RQ1 Can a fine-tuned BERT model effectively extract credit risk signals from unstructured loan descriptions in P2P lending?
• RQ2 How does the BERT-generated risk score compare to traditional credit variables in predicting loan default?
• RQ3 To what extent do linguistic factors such as tone, clarity, and structure influence the BERT risk score?
• RQ4 What are the implications of model opacity and potential biases for regulatory compliance and user trust?
• RQ5 Can alternative LLM architectures or prompting strategies improve performance while increasing interpretability?

Key findings

• The BERT-based risk score significantly improves the performance of a XGBoost credit risk classifier, outperforming models using only categorical variables.
• The BERT score alone achieved superior classification performance compared to models relying solely on demographic and financial features.
• Loan descriptions with higher linguistic quality—clarity, confidence, and objectivity—tended to receive lower risk scores, suggesting a bias toward well-written narratives.
• The model’s risk scores were consistently below 0.4 for loans with detailed, objective, and confident descriptions, aligning with non-default predictions.
• Despite strong performance, the black-box nature of BERT limits interpretability, and the exact drivers behind individual scores remain opaque.
• The study identifies a need for more transparent methods, such as hybrid topic modeling with embeddings, to improve explainability in LLM-based risk assessment.