[Paper Review] Capturing Failures of Large Language Models via Human Cognitive Biases

Erik Jones, Jacob Steinhardt|arXiv (Cornell University)|Feb 24, 2022

Topic Modeling36 citations

TL;DR

The paper presents a cognitive-bias–inspired framework to elicit qualitative failure modes in large language models, applies it to Codex, CodeGen, and GPT-3, and demonstrates high-impact errors such as erroneous file deletions.

ABSTRACT

Large language models generate complex, open-ended outputs: instead of outputting a class label they write summaries, generate dialogue, or produce working code. In order to asses the reliability of these open-ended generation systems, we aim to identify qualitative categories of erroneous behavior, beyond identifying individual errors. To hypothesize and test for such qualitative errors, we draw inspiration from human cognitive biases -- systematic patterns of deviation from rational judgement. Specifically, we use cognitive biases as motivation to (i) generate hypotheses for problems that models may have, and (ii) develop experiments that elicit these problems. Using code generation as a case study, we find that OpenAI's Codex errs predictably based on how the input prompt is framed, adjusts outputs towards anchors, and is biased towards outputs that mimic frequent training examples. We then use our framework to elicit high-impact errors such as incorrectly deleting files. Our results indicate that experimental methodology from cognitive science can help characterize how machine learning systems behave.

Motivation & Objective

• Motivate the study by examining reliability challenges in open-ended generation systems like code and language models.
• Introduce a cognitive-bias–based methodology to hypothesize and test qualitative error modes.
• Demonstrate the framework by eliciting and characterizing failure modes across Codex, CodeGen, and GPT-3.
• Show that framing, anchoring, availability, and attribute substitution biases can trigger high-impact errors.

Proposed method

• Translate cognitive biases into prompt transformations that preserve semantics but trigger targeted failures.
• Measure model sensitivity by the drop in functional accuracy after transformations.
• Check that outputs exhibit explicit indicators of the targeted failure (e.g., reliance on irrelevant information, anchoring effects).
• Apply framing, anchoring, availability heuristic, and attribute substitution to codemodels and GPT-3 to elicit specific errors.
• Use a black-box prompt-based approach; no access to training data or model parameters required.

Experimental results

Research questions

• RQ1 Can cognitive biases inspire hypotheses about qualitative failure modes in open-ended generation models?
• RQ2 Do prompt transformations inspired by framing, anchoring, availability, and attribute substitution systematically reveal specific failures in Codex, CodeGen, and GPT-3?
• RQ3 Are high-impact errors (e.g., destructive code behavior) uncoverable through this framework?

Key findings

• Code generation models (Codex and CodeGen) rely on irrelevant prompt information when framed, anchored, or presented with related but incorrect prompts.
• Anchoring and availability-inspired prompts cause outputs to shift toward related solutions or training-data-frequency patterns, reducing accuracy.
• Attribute substitution prompts reveal simple heuristics (e.g., using function names) that bias generated solutions.
• GPT-3 shows qualitative framing and anchoring effects similar to humans, evidencing predictable shifts in estimates or choices under framing.
• The framework can systematically generate high-impact errors, such as file deletions in Codex, highlighting the potential for dangerous misbehaviors.
• Overall, cognitive-science–based experimental methodology provides a practical, model-agnostic way to characterize qualitative failures in open-ended ML systems.