[Paper Review] BrainBench: Exposing the Commonsense Reasoning Gap in Large Language Models

Yuzhe Tang|arXiv (Cornell University)|Mar 16, 2026

Artificial Intelligence in Healthcare and Education0 citations

TL;DR

BrainBench introduces 100 brainteaser questions across 20 categories to diagnose commonsense reasoning gaps in LLMs, revealing surface-heuristic failures and variability across Claude and GPT models. Extended thinking helps some categories but harms others; cross-lingual results in Chinese show similar deficits.

ABSTRACT

Large language models (LLMs) achieve impressive scores on standard benchmarks yet routinely fail questions that any human would answer correctly in seconds. We introduce BrainBench, a benchmark of 100 brainteaser questions spanning 20 carefully designed categories, each targeting a specific commonsense reasoning failure mode in LLMs. Categories range from implicit physical constraints ("Should I walk or drive my rental car to the return lot?") to semantic scope tricks and default assumption hijacks. We evaluate eight frontier models -- four from the Claude family and four from the GPT family -- using a zero-shot protocol with 10 independent runs per question. The best model, Claude Opus 4.6 with extended thinking, achieves only 80.3% accuracy; the worst, GPT-4o, scores 39.7%. Even top-performing models exhibit a 6-16 percentage-point gap between accuracy and consistency, revealing stochastic reasoning. Cross-lingual evaluation in Chinese shows most models degrade by 2-8 percentage points, confirming that these failures reflect reasoning deficits rather than language-specific artifacts. BrainBench provides a fine-grained diagnostic tool for identifying where and why LLMs substitute surface heuristics for genuine commonsense reasoning.

Motivation & Objective

• Develop a taxonomy of 20 commonsense reasoning failure categories that expose how LLMs rely on surface heuristics.
• Create a 100-question benchmark designed to be easy for humans but challenging for LLMs across targeted failure modes.
• Evaluate eight frontier models (Claude and GPT families) under a zero-shot protocol with multiple runs to measure accuracy and consistency.
• Assess cross-lingual robustness by evaluating English and Chinese versions of BrainBench.

Proposed method

• Define 20 failure-category taxonomy with core trap, rationale, and correct reasoning.
• Design 100 questions (5 per category) that are trivially easy for humans but trap LLMs.
• Use a strict zero-shot protocol with 10 independent runs per question per model (300 responses per model).
• Employ an LLM judge to determine correctness by ground-truth reasoning, not phrasing.
• Measure two metrics: accuracy (fraction correct across runs) and consistency (fraction correct in all 10 runs).
• Translate the benchmark to Chinese and evaluate under the same protocol to test cross-lingual robustness.

Experimental results

Research questions

• RQ1 What commonsense reasoning failure modes most reliably distinguish frontier LLMs from humans?
• RQ2 How do different model families (Claude vs GPT) compare in accuracy, consistency, and response stability on BrainBench?
• RQ3 Does extended thinking improve or degrade performance across categories?
• RQ4 Is the observed reasoning gap language-specific or language-agnostic?

Key findings

• Claude models achieve 74–80% accuracy, GPT-5.4 models 70–74%, and GPT-4o around 40% accuracy.
• Hardest categories (implicit physical constraint and wrong vantage point) average 40% accuracy across models.
• Extended thinking yields about +3 pp overall but can hurt certain categories, indicating an overthinking paradox.
• Cross-lingual evaluation shows English→Chinese average drop of 2.6 pp, with Claude Opus 4.6 sometimes improving in Chinese.
• There is a 31.0 pp jump from GPT-4o to GPT-5.4 within the GPT family, while Claude models show smaller gains across variants.
• A notable consistency gap (accuracy minus consistency) exists, e.g., Claude Opus 4.6 Think has 80.3% accuracy but 74.0% consistency.