[Paper Review] Competitive Programming with Large Reasoning Models

OpenAI, :+24 more|ArXiv.org|Feb 3, 2025

Multi-Criteria Decision Making5 citations

TL;DR

Reinforcement-learning–driven large reasoning models (o1, o1-ioi, o3) substantially improve competitive programming performance, surpass domain-specific hand-crafted strategies, and achieve gold at IOI 2024 and top Codeforces standings.

ABSTRACT

We show that reinforcement learning applied to large language models (LLMs) significantly boosts performance on complex coding and reasoning tasks. Additionally, we compare two general-purpose reasoning models - OpenAI o1 and an early checkpoint of o3 - with a domain-specific system, o1-ioi, which uses hand-engineered inference strategies designed for competing in the 2024 International Olympiad in Informatics (IOI). We competed live at IOI 2024 with o1-ioi and, using hand-crafted test-time strategies, placed in the 49th percentile. Under relaxed competition constraints, o1-ioi achieved a gold medal. However, when evaluating later models such as o3, we find that o3 achieves gold without hand-crafted domain-specific strategies or relaxed constraints. Our findings show that although specialized pipelines such as o1-ioi yield solid improvements, the scaled-up, general-purpose o3 model surpasses those results without relying on hand-crafted inference heuristics. Notably, o3 achieves a gold medal at the 2024 IOI and obtains a Codeforces rating on par with elite human competitors. Overall, these results indicate that scaling general-purpose reinforcement learning, rather than relying on domain-specific techniques, offers a robust path toward state-of-the-art AI in reasoning domains, such as competitive programming.

Motivation & Objective

• Motivate the use of large reasoning models to tackle complex, objectively gradable coding problems.
• Compare general-purpose RL-powered models with domain-specific, hand-engineered inference strategies.
• Evaluate performance across multiple competitive programming benchmarks (CodeForces, IOI) and real-world software tasks.
• Assess how scaling RL training versus relying on human-crafted test-time heuristics affects state-of-the-art results.

Proposed method

• Train OpenAI o1 with reinforcement learning to enhance coding and reasoning abilities and enable code execution during inference.
• Fine-tune a variant, o1-ioi, with additional coding-focused RL and domain-specific test-time strategies tailored for IOI-style tasks.
• Evaluate by simulating CodeForces contests under official-like constraints and conducting contamination checks using embeddings.
• Compare with early checkpoints of OpenAI o3 to study end-to-end RL without hand-crafted test-time heuristics.
• Analyze performance on IOI 2024 problems under official rules and under relaxed submission limits to isolate the effect of test-time strategies.
• Assess real-world coding tasks with SWE-bench verified and HackerRank Astra to test generalization of reasoning capabilities.

Experimental results

Research questions

• RQ1 Does scaling reinforcement learning improve coding and reasoning performance beyond hand-engineered inference pipelines?
• RQ2 How do domain-specific test-time strategies compare to end-to-end RL in competitive programming tasks?
• RQ3 Can a large reasoning model like o3 achieve gold-level performance under standard contest constraints without hand-crafted heuristics?
• RQ4 Do improvements in competitive programming translate to real-world software engineering benchmarks?

Key findings

• o1-ioi improves over o1 with additional RL and IOI-specific test-time strategies, achieving a CodeForces rating of 1807 (62nd percentile) and 2214 (98th percentile) with full test-time strategy.
• o3, trained with much larger RL compute, surpasses o1-ioi, reaching a CodeForces rating of 2724 (99.8th percentile) and 395.64 IOI points under standard submission limits (gold threshold ~360).
• At IOI 2024 live, o1-ioi scored 213 points (49th percentile) with 50 submissions; under relaxed limits, 362.14 points would exceed the gold threshold.
• o3 demonstrates autonomous self-improvement in test-time reasoning (e.g., sometimes generating brute-force verifications to validate outputs) without human-engineered heuristics.
• In SWE-bench and Astra real-world coding tasks, reasoning-focused models show meaningful gains: o1-preview improves over GPT-4o in pass@1 by ~9.98 percentage points, o1 after RL improves to 63.92% pass@1, and o3 shows a 22.8% improvement over o1 on SWE-bench.