[Paper Review] Recursion of Thought: A Divide-and-Conquer Approach to Multi-Context Reasoning with Language Models

Soochan Lee, Gunhee Kim|arXiv (Cornell University)|Jun 12, 2023

Topic Modeling4 citations

TL;DR

This paper introduces Recursion of Thought (RoT), a model-agnostic inference framework that enables large language models to solve complex reasoning problems exceeding their context length by recursively dividing tasks into multiple sub-contexts using special tokens (GO, STOP, THINK). RoT enables models like GPT-3 to solve problems requiring hundreds of thousands of reasoning tokens—demonstrating near-perfect accuracy on arithmetic and algorithmic tasks far beyond standard context limits.

ABSTRACT

Generating intermediate steps, or Chain of Thought (CoT), is an effective way to significantly improve language models' (LM) multi-step reasoning capability. However, the CoT lengths can grow rapidly with the problem complexity, easily exceeding the maximum context size. Instead of increasing the context limit, which has already been heavily investigated, we explore an orthogonal direction: making LMs divide a problem into multiple contexts. We propose a new inference framework, called Recursion of Thought (RoT), which introduces several special tokens that the models can output to trigger context-related operations. Extensive experiments with multiple architectures including GPT-3 show that RoT dramatically improves LMs' inference capability to solve problems, whose solution consists of hundreds of thousands of tokens.

Motivation & Objective

• To address the critical limitation of fixed context length in large language models when solving complex, multi-step reasoning problems.
• To enable language models to solve reasoning tasks whose solution requires hundreds of thousands of tokens, exceeding standard context windows.
• To propose a model-agnostic inference framework that does not require architectural changes or task-specific tools.
• To explore a divide-and-conquer strategy for reasoning that scales beyond the constraints of single-context generation.

Proposed method

• Introduces three special tokens: GO (start of a problem), STOP (end of a problem), and THINK (triggers recursion into a new context).
• Structures each inference context as a concatenation of main and sub-problem questions and answers, with nested GO-STOP pairs for hierarchical decomposition.
• Uses the THINK token to initiate a recursive inference process, creating a new context for each sub-problem while preserving the main problem's context.
• Employs a recursive generation strategy where sub-problems are solved in isolation and their answers are integrated into the main solution flow.
• Designs a synthetic benchmark with eight arithmetic and algorithmic tasks, adjustable to extreme difficulty (e.g., 64-digit arithmetic), to evaluate long-context reasoning.
• Trains models end-to-end to generate these special tokens strategically, enabling scalable, multi-context reasoning without external tools.

Experimental results

Research questions

• RQ1 Can language models solve reasoning problems whose solution length vastly exceeds their context window using only self-generated special tokens?
• RQ2 How effective is the recursive decomposition strategy (via THINK tokens) in enabling models to handle problems with 100K+ token reasoning steps?
• RQ3 Does the RoT framework outperform standard Chain-of-Thought (CoT) and baseline prompting (WT) on ultra-long reasoning tasks?
• RQ4 Can RoT generalize across diverse reasoning types, including arithmetic, dynamic programming, and string operations, without task-specific adaptation?
• RQ5 What is the performance ceiling of RoT on extremely long problems, and how does it scale with model size and architecture?

Key findings

• RoT achieves 100% accuracy on 8-digit and 16-digit addition and subtraction tasks, even when the reasoning steps exceed 100K tokens.
• For 64-digit addition and subtraction, RoT maintains 100% accuracy, while standard CoT fails completely (0% accuracy), demonstrating its scalability.
• On multiplication and division tasks, RoT achieves 99.9% accuracy on 32-digit problems, significantly outperforming CoT and WT baselines.
• For algorithmic problems like Longest Common Subsequence (LCS) and Longest Palindromic Subsequence (LPS), RoT maintains 100% accuracy up to length 32, while CoT fails beyond length 12.
• On the 0-1 Knapsack and Matrix Chain Multiplication (MCM) problems, RoT achieves 100% accuracy on small instances and maintains high performance (e.g., 99.8% on LPS with 56 elements), far exceeding baselines.
• Even with LSTM-based models, RoT achieves 98.6% accuracy on 12-digit addition, while CoT fails entirely, showing robustness across architectures.