[Paper Review] Cognitive Bias in Decision-Making with LLMs

Jessica Maria Echterhoff, Yao Liu+3 more|arXiv (Cornell University)|Feb 25, 2024

Big Data and Business Intelligence12 citations

TL;DR

BiasBuster framework to detect and mitigate cognitive bias in LLMs for high-stakes decisions, with a 16,800-prompt dataset and self-debiasing prompts.

ABSTRACT

Large language models (LLMs) offer significant potential as tools to support an expanding range of decision-making tasks. Given their training on human (created) data, LLMs have been shown to inherit societal biases against protected groups, as well as be subject to bias functionally resembling cognitive bias. Human-like bias can impede fair and explainable decisions made with LLM assistance. Our work introduces BiasBuster, a framework designed to uncover, evaluate, and mitigate cognitive bias in LLMs, particularly in high-stakes decision-making tasks. Inspired by prior research in psychology and cognitive science, we develop a dataset containing 13,465 prompts to evaluate LLM decisions on different cognitive biases (e.g., prompt-induced, sequential, inherent). We test various bias mitigation strategies, while proposing a novel method utilizing LLMs to debias their own human-like cognitive bias within prompts. Our analysis provides a comprehensive picture of the presence and effects of cognitive bias across commercial and open-source models. We demonstrate that our selfhelp debiasing effectively mitigates model answers that display patterns akin to human cognitive bias without having to manually craft examples for each bias.

Motivation & Objective

• Motivate the need to audit cognitive bias in LLM-assisted high-stakes decision making and to distinguish it from societal bias.
• Develop a cognitive bias testing framework (BiasBuster) grounded in cognitive science to quantify bias in LLMs.
• Create an extensive 16,800-prompt dataset to evaluate prompt-induced, sequential, and inherent biases across models.
• Evaluate various bias mitigation strategies, including zero-shot, few-shot, and self-debiasing (self-help) methods.
• Demonstrate that self-help debiasing can reduce cognitive bias without manually crafting bias-specific examples.

Proposed method

• Define cognitive bias categories (prompt-based, inherent, sequential) and map them to test prompts inspired by cognitive science experiments.
• Develop a 16,800-prompt dataset covering anchoring, status quo, framing, and group attribution biases.
• Quantify bias via metrics such as decision distributions, confidence, and Euclidean distance between expected and observed choices.
• Compare mitigation strategies: zero-shot, few-shot (contrastive/counterfactual), and self-help prompting where the model rewrites its own prompts.
• Evaluate across multiple models (GPT-3.5-turbo, GPT-4, Llama-2 7B/13B) on high-stakes student-admission style tasks.
• Propose a self-help procedure where the model rewrites prompts to de-bias its own responses and assess its impact.

Experimental results

Research questions

• RQ1 Do LLMs exhibit prompt-induced, sequential, and inherent cognitive biases in high-stakes decision tasks?
• RQ2 How do different bias mitigation strategies (zero-shot, few-shot, self-help) affect bias prevalence and decision quality across models?
• RQ3 Can LLMs effectively debias their own prompts (self-help) without manually crafted bias examples?
• RQ4 How do model capacity and type influence susceptibility to cognitive bias and effectiveness of debiasing?
• RQ5 What is the comparative impact of framing, anchoring, primacy, status quo, and group attribution biases on admission decisions?

Key findings

• Cognitive biases (framing and group attribution) are present across models, with GPT-4 showing framing bias and Llama-2-7B showing group attribution bias.
• Few-shot mitigation can create failure cases and may drastically alter outputs in open-source models.
• Self-help debiasing improves decision consistency for high-capacity models and can reduce bias-inducing prompts to zero in some cases; effects vary by model.
• Self-help reduces primacy/anchoring effects for higher-capacity models, but lower-capacity models show limited improvement.
• Model capacity strongly influences the effectiveness of self-debiasing; higher-capacity models achieve greater debiasing success.
• The BiasBuster dataset enables systematic testing of cognitive biases and mitigation strategies in LLMs.