[Paper Review] Are Language Models Worse than Humans at Following Prompts? It's Complicated

Albert Webson, Alyssa Marie Loo+2 more|arXiv (Cornell University)|Jan 17, 2023

Topic Modeling4 citations

TL;DR

This paper challenges the assumption that language models (LMs) fail to follow prompts like humans do, by empirically testing human behavior under the same pathological prompts used in prior LM studies. It finds that while humans ignore irrelevant prompts (like LMs), they follow misleading prompts faithfully—contrary to the assumption that humans would reject such instructions, revealing a key divergence in human vs. model behavior under deception.

ABSTRACT

Prompts have been the center of progress in advancing language models' zero-shot and few-shot performance. However, recent work finds that models can perform surprisingly well when given intentionally irrelevant or misleading prompts. Such results may be interpreted as evidence that model behavior is not "human like". In this study, we challenge a central assumption in such work: that humans would perform badly when given pathological instructions. We find that humans are able to reliably ignore irrelevant instructions and thus, like models, perform well on the underlying task despite an apparent lack of signal regarding the task they are being asked to do. However, when given deliberately misleading instructions, humans follow the instructions faithfully, whereas models do not. Our findings caution that future research should not idealize human behaviors as a monolith and should not train or evaluate models to mimic assumptions about these behaviors without first validating humans' behaviors empirically.

Motivation & Objective

• To empirically test whether humans behave as assumed in prior studies that claim language models fail to follow prompts like humans do.
• To investigate whether humans reliably ignore irrelevant or misleading prompts, as assumed in evaluations of language model behavior.
• To challenge the assumption that human-like prompt following is a valid benchmark for evaluating language models.
• To highlight the need for empirical validation of human behavior before using it as a standard for model evaluation.

Proposed method

• Adapted the experimental setup from Webson & Pavlick (2022) to test human performance on the same NLI and coreference resolution tasks with instructive, misleading, irrelevant, and null prompts.
• Conducted zero-shot and few-shot experiments with human participants, including controls for prior NLP experience.
• Used identical prompt categories (instructive, misleading, irrelevant, null) as in prior LM studies to ensure direct comparison.
• Collected free-text responses to understand participants’ reasoning and interpretation of prompts.
• Analyzed performance across prompt types, comparing human accuracy to model performance on the same stimuli.
• Applied statistical analysis to assess whether human performance varied meaningfully across prompt categories.

Experimental results

Research questions

• RQ1 Do humans perform worse on NLI tasks when given misleading prompts, as assumed in prior LM evaluations?
• RQ2 Can humans reliably ignore irrelevant prompts, similar to language models?
• RQ3 Do humans interpret and follow misleading prompts in the same way as language models, or do they instead reject them as nonsensical?
• RQ4 How does prior exposure to NLP tasks affect human performance on pathological prompts?
• RQ5 To what extent do human interpretations of prompts align with the intended task, versus surface-level lexical cues?

Key findings

• Humans perform nearly as well on NLI tasks with irrelevant prompts as they do with instructive prompts, indicating they can ignore irrelevant content.
• Humans perform significantly worse on NLI tasks when given misleading prompts—contrary to the assumption that they would reject such prompts as nonsensical.
• Humans follow misleading prompts faithfully when the surface instruction suggests a different task (e.g., grammaticality), even when the actual task is NLI.
• Performance on misleading prompts is strongly correlated with participants’ interpretation of the surface task, not the underlying NLI logic.
• Participants with no prior NLP experience still performed poorly on misleading prompts, indicating the issue is not due to domain-specific knowledge.
• Free-text responses show that many participants interpreted misleading prompts as asking for a different task (e.g., grammaticality or paraphrasing), leading them to give incorrect answers on the actual NLI task.