[论文解读] BrainBench: Exposing the Commonsense Reasoning Gap in Large Language Models

Yuzhe Tang|arXiv (Cornell University)|Mar 16, 2026

Artificial Intelligence in Healthcare and Education被引用 0

一句话总结

BrainBench 以 100 道跨 20 类失败类型的问题来诊断 LLM 的常识推理缺口，揭示表面启发式失效以及在 Claude 与 GPT 模型之间的变异性。扩展思考对某些类别有帮助但对其他类别有害；中文的跨语言结果显示出类似的缺陷。

ABSTRACT

Large language models (LLMs) achieve impressive scores on standard benchmarks yet routinely fail questions that any human would answer correctly in seconds. We introduce BrainBench, a benchmark of 100 brainteaser questions spanning 20 carefully designed categories, each targeting a specific commonsense reasoning failure mode in LLMs. Categories range from implicit physical constraints ("Should I walk or drive my rental car to the return lot?") to semantic scope tricks and default assumption hijacks. We evaluate eight frontier models -- four from the Claude family and four from the GPT family -- using a zero-shot protocol with 10 independent runs per question. The best model, Claude Opus 4.6 with extended thinking, achieves only 80.3% accuracy; the worst, GPT-4o, scores 39.7%. Even top-performing models exhibit a 6-16 percentage-point gap between accuracy and consistency, revealing stochastic reasoning. Cross-lingual evaluation in Chinese shows most models degrade by 2-8 percentage points, confirming that these failures reflect reasoning deficits rather than language-specific artifacts. BrainBench provides a fine-grained diagnostic tool for identifying where and why LLMs substitute surface heuristics for genuine commonsense reasoning.

研究动机与目标

• Develop a taxonomy of 20 commonsense reasoning failure categories that expose how LLMs rely on surface heuristics.
• Create a 100-question benchmark designed to be easy for humans but challenging for LLMs across targeted failure modes.
• Evaluate eight frontier models (Claude and GPT families) under a zero-shot protocol with multiple runs to measure accuracy and consistency.
• Assess cross-lingual robustness by evaluating English and Chinese versions of BrainBench.

提出的方法

• Define 20 failure-category taxonomy with core trap, rationale, and correct reasoning.
• Design 100 questions (5 per category) that are trivially easy for humans but trap LLMs.
• Use a strict zero-shot protocol with 10 independent runs per question per model (300 responses per model).
• Employ an LLM judge to determine correctness by ground-truth reasoning, not phrasing.
• Measure two metrics: accuracy (fraction correct across runs) and consistency (fraction correct in all 10 runs).
• Translate the benchmark to Chinese and evaluate under the same protocol to test cross-lingual robustness.

实验结果

研究问题

• RQ1 What commonsense reasoning failure modes most reliably distinguish frontier LLMs from humans?
• RQ2 How do different model families (Claude vs GPT) compare in accuracy, consistency, and response stability on BrainBench?
• RQ3 Does extended thinking improve or degrade performance across categories?
• RQ4 Is the observed reasoning gap language-specific or language-agnostic?

主要发现

• Claude models achieve 74–80% accuracy, GPT-5.4 models 70–74%, and GPT-4o around 40% accuracy.
• Hardest categories (implicit physical constraint and wrong vantage point) average 40% accuracy across models.
• Extended thinking yields about +3 pp overall but can hurt certain categories, indicating an overthinking paradox.
• Cross-lingual evaluation shows English→Chinese average drop of 2.6 pp, with Claude Opus 4.6 sometimes improving in Chinese.
• There is a 31.0 pp jump from GPT-4o to GPT-5.4 within the GPT family, while Claude models show smaller gains across variants.
• A notable consistency gap (accuracy minus consistency) exists, e.g., Claude Opus 4.6 Think has 80.3% accuracy but 74.0% consistency.