[Paper Review] EsoLang-Bench: Evaluating Genuine Reasoning in Large Language Models via Esoteric Programming Languages

Aman Sharma, Paras Chopra|arXiv (Cornell University)|Mar 10, 2026

Machine Learning in Materials Science0 citations

TL;DR

EsoLang-Bench introduces a five-language esoteric benchmark to test genuine reasoning in LLMs, showing large gaps between standard benchmark performance and out-of-distribution esoteric tasks, even with advanced prompting and agentic systems.

ABSTRACT

Large language models achieve near-ceiling performance on code generation benchmarks, yet these results increasingly reflect memorization rather than genuine reasoning. We introduce EsoLang-Bench, a benchmark using five esoteric programming languages (Brainfuck, Befunge-98, Whitespace, Unlambda, and Shakespeare) that lack benchmark gaming incentives due to their economic irrationality for pre-training. These languages require the same computational primitives as mainstream programming but have 1,000-100,000x fewer public repositories than Python (based on GitHub search counts). We evaluate five frontier models across five prompting strategies and find a dramatic capability gap: models achieving 85-95% on standard benchmarks score only 0-11% on equivalent esoteric tasks, with 0% accuracy beyond the Easy tier. Few-shot learning and self-reflection fail to improve performance, suggesting these techniques exploit training priors rather than enabling genuine learning. EsoLang-Bench provides the first benchmark designed to mimic human learning by acquiring new languages through documentation, interpreter feedback, and iterative experimentation, measuring transferable reasoning skills resistant to data contamination.

Motivation & Objective

• Motivate evaluation of genuine reasoning by using languages with minimal pre-training data to avoid benchmarking gaming.
• Provide a dataset of 80 problems across four difficulty levels implemented in five esoteric languages.
• Assess the impact of prompting strategies and agentic tool usage on out-of-distribution (OOD) reasoning performance.
• Analyze error modes (compile, runtime, logic) to understand fundamental limitations in current frontier models.

Proposed method

• Create EsoLang-Bench with 80 problems across Easy, Medium, Hard, and Extra-Hard levels implemented in Brainfuck, Befunge-98, Whitespace, Unlambda, and Shakespeare.
• Evaluate five frontier models under five prompting strategies (Zero-Shot, Few-Shot, Self-Scaffolding, Textual Self-Scaffolding, ReAct).
• Test agentic systems (Codex, Claude Code) with interpreter feedback loops to study the impact of tool use on OOD tasks.
• Use automated interpreter-based verification with 6 test cases per problem and bootstrap statistics (three seeds; 95% CI).
• Analyze errors by categorizing compile, runtime, and logic failures to identify core bottlenecks.

Experimental results

Research questions

• RQ1 Can LLMs transfer computational reasoning to esoteric languages with minimal pre-training data exposure?
• RQ2 Do in-context learning and advanced prompting improve OOD esoteric-language performance, or is pre-training coverage the limiting factor?
• RQ3 Do agentic systems with interpreters and feedback loops outperform non-agentic baselines on these OOD tasks?
• RQ4 What are the predominant error modes (compile, runtime, logic) when solving esoteric-language problems, and what do they reveal about model limitations?

Key findings

• Frontier models show a dramatic capability gap: high performance on standard benchmarks does not translate to esoteric-language tasks (0–11% average accuracy on esoteric tasks).
• All models score 0% on Medium, Hard, and Extra-Hard esoteric problems across configurations; only Easy-tier problems see any success.
• Self-scaffolding offers the best non-agentic prompting result, with up to about 11.2% accuracy in some language/strategy combos, but remains far below Python performance.
• Few-shot prompting yields negligible gains in this ultra-low-resource setting, supporting the hypothesis that ICL effectiveness hinges on pre-training data coverage.
• Agentic systems with interpreter access provide 2–3x improvements over non-agentic baselines, yet still remain far from parity with mainstream languages.
• Error analysis shows compilation errors dominate in many languages, with semantic (logic) errors prevalent where syntax is acquired, and near-total compilation failure in the most data-scarce languages like Whitespace and Unlambda.