[Paper Review] Do Language Models Exhibit the Same Cognitive Biases in Problem Solving as Human Learners?

Andreas Opedal, Alessandro Stolfo+6 more|arXiv (Cornell University)|Jan 31, 2024

Education and Critical Thinking Development21 citations

TL;DR

The paper investigates whether current LLMs exhibit human-like biases in solving arithmetic word problems, identifying biases at three solving stages and testing multiple open-source models with instruction-tuning.

ABSTRACT

There is increasing interest in employing large language models (LLMs) as cognitive models. For such purposes, it is central to understand which properties of human cognition are well-modeled by LLMs, and which are not. In this work, we study the biases of LLMs in relation to those known in children when solving arithmetic word problems. Surveying the learning science literature, we posit that the problem-solving process can be split into three distinct steps: text comprehension, solution planning and solution execution. We construct tests for each one in order to understand whether current LLMs display the same cognitive biases as children in these steps. We generate a novel set of word problems for each of these tests, using a neuro-symbolic approach that enables fine-grained control over the problem features. We find evidence that LLMs, with and without instruction-tuning, exhibit human-like biases in both the text-comprehension and the solution-planning steps of the solving process, but not in the final step, in which the arithmetic expressions are executed to obtain the answer.

Motivation & Objective

• Motivate the use of LLMs as cognitive models for human learning and identify where they align with or diverge from human biases in arithmetic word problem solving.
• Develop a controlled problem generation pipeline to test specific biases at three solving stages (text comprehension, solution planning, solution execution).
• Empirically evaluate open-source LLMs (LLaMA2, Mistral, Mixtral) with and without instruction-tuning across multiple prompting regimes to detect bias patterns.
• Quantify causal effects of targeted problem features on model performance via conditional average treatment effect (CATE) estimates.

Proposed method

• Propose a three-step cognitive model of problem solving (text comprehension, solution planning, solution execution) and operationalize it with MathWorld logical forms and a symbolic-expression proof sequence.
• Generate a controlled dataset of arithmetic word problems using a neuro-symbolic pipeline that fixes problem structure, instantiates mental models, renders templated text, and applies a post-editing error-correction step.
• Use pairwise problem generation to create x and x' variants for a chosen feature, enabling causal estimation of a feature's effect on model accuracy via CATE.
• Evaluate eight model configurations (LLaMA2 7B/13B, Mistral 7B, Mixtral 8x7B, with and without instruction-tuning) under direct prompting and chain-of-thought prompting, using zero-shot inferences.
• Apply statistical tests (paired t-tests) to determine whether observed CATEs differ from zero and report p-values where applicable.

Experimental results

Research questions

• RQ1 Do LLMs show consistency bias in problem text where relational keywords align with the required operation?
• RQ2 Do LLMs exhibit transfer vs. comparison bias at the mental-model level when solving problems?
• RQ3 Do LLMs display carry effects during the symbolic-expression execution step, especially with numbers that induce carries?
• RQ4 How do instruction-tuned versus non-tuned models compare in exhibiting these biases across prompting regimes (direct vs. chain-of-thought)?

Key findings

• LLMs exhibit human-like consistency bias at the problem text level, with lower accuracy on inconsistent statements compared to consistent ones.
• Transfer vs. comparison bias is present in LLMs, mirroring child learners, across multiple models and prompting settings.
• Carry effects are not consistently observed in the solution-execution step across tested models and prompting methods.
• Chain-of-thought prompting can amplify certain biases (e.g., consistency bias) but improves overall performance, depending on the model and task setup.
• Instruction-tuned models generally show larger CATEs for certain biases compared to pretrained-only variants, depending on the prompt regime.
• Across models and tests, several biases reach statistical significance (p-values often < 0.01 for key comparisons).