[Paper Review] Fairness of ChatGPT and the Role Of Explainable-Guided Prompts

Yashar Deldjoo|arXiv (Cornell University)|Jul 14, 2023

Artificial Intelligence in Healthcare and Education13 citations

TL;DR

The paper investigates using OpenAI’s GPT with prompt engineering and domain knowledge for credit risk assessment, showing competitive results with traditional ML using far less data and offering insights on fairness across genders.

ABSTRACT

Our research investigates the potential of Large-scale Language Models (LLMs), specifically OpenAI's GPT, in credit risk assessment-a binary classification task. Our findings suggest that LLMs, when directed by judiciously designed prompts and supplemented with domain-specific knowledge, can parallel the performance of traditional Machine Learning (ML) models. Intriguingly, they achieve this with significantly less data-40 times less, utilizing merely 20 data points compared to the ML's 800. LLMs particularly excel in minimizing false positives and enhancing fairness, both being vital aspects of risk analysis. While our results did not surpass those of classical ML models, they underscore the potential of LLMs in analogous tasks, laying a groundwork for future explorations into harnessing the capabilities of LLMs in diverse ML tasks.

Motivation & Objective

• Investigate whether LLMs like ChatGPT can perform credit risk classification with limited data.
• Assess how prompt design and domain knowledge influence prediction accuracy.
• Examine gender fairness using bootstrap-based TPR comparisons and statistical tests.
• Provide guidelines for integrating domain knowledge and prompt strategies to improve ML tasks with LLMs.

Proposed method

• Convert the credit risk task into a chat-based prediction problem using ChatGPT-3.5-Turbo.
• Design multi-part prompts including task instruction, in-context examples, attribute descriptions, domain knowledge, and problem formulation.
• Incorporate domain knowledge via ML-derived feature importance (MLFI) and its ordered variant (MLFI-ord).
• Evaluate performance using classical ML models and OpenAI-based models on the German Credit dataset with 5-fold cross-validation.
• Use bootstrap sampling (1000 resamples) to assess gender fairness through true-positive rate disparities.
• Compare accuracy, precision, recall, F1, and costs for false positives/negatives across models.

Figure 1 : Flowchart illustrating the conceptual framework of the paper

Experimental results

Research questions

• RQ1 Can ChatGPT with carefully crafted prompts and domain knowledge achieve competitive credit risk predictions with far less data than classical ML?
• RQ2 How does prompt design (including domain knowledge and feature ordering) affect accuracy and fairness in OpenAI-based predictions?
• RQ3 Do OpenAI-based models exhibit different gender fairness characteristics compared to traditional ML models in credit risk assessment?
• RQ4 What practical guidelines emerge for integrating prompt engineering and domain knowledge to improve ML tasks with LLMs?

Key findings

• OpenAI-based models achieve competitive results with only 20 training samples versus 800 for classical ML models.
• AdaBoost with MLFI prompts yields high precision, recall, and F1 (0.7305) among OpenAI-based configurations.
• Classical ML models outperform OpenAI-based models on average accuracy metrics (0.7792 vs 0.7129 accuracy; 0.8822 vs 0.6078 recall; 0.8302 vs 0.6528 F1).
• OpenAI models show lower false-positive costs compared to classical ML models, indicating a more cautious credit approval tendency.
• Fairness analyses reveal that certain prompts (e.g., Prompt-5, Prompt-7) approach non-significant gender disparities, while others show significant differences; classical models generally reject the fairness null hypothesis (significant disparity).
• Prompts can sometimes promote fairness in outcomes, highlighting the potential of prompt-based fairness considerations in LLM-assisted ML tasks.