[Paper Review] Large Language Model Guided Tree-of-Thought

Jieyi Long|arXiv (Cornell University)|May 15, 2023

graph theory and CDMA systemsEngineering51 citations

TL;DR

Introduces the Tree-of-Thought (ToT) framework that augments an LLM with a prompter, checker, memory, and ToT controller to enable backtracking in multi-round problem solving; demonstrates improved Sudoku solving performance.

ABSTRACT

In this paper, we introduce the Tree-of-Thought (ToT) framework, a novel approach aimed at improving the problem-solving capabilities of auto-regressive large language models (LLMs). The ToT technique is inspired by the human mind's approach for solving complex reasoning tasks through trial and error. In this process, the human mind explores the solution space through a tree-like thought process, allowing for backtracking when necessary. To implement ToT as a software system, we augment an LLM with additional modules including a prompter agent, a checker module, a memory module, and a ToT controller. In order to solve a given problem, these modules engage in a multi-round conversation with the LLM. The memory module records the conversation and state history of the problem solving process, which allows the system to backtrack to the previous steps of the thought-process and explore other directions from there. To verify the effectiveness of the proposed technique, we implemented a ToT-based solver for the Sudoku Puzzle. Experimental results show that the ToT framework can significantly increase the success rate of Sudoku puzzle solving. Our implementation of the ToT-based Sudoku solver is available on GitHub: \url{https://github.com/jieyilong/tree-of-thought-puzzle-solver}.

Motivation & Objective

• Motivate the need for long-range reasoning in LLMs and address failures of linear generation and lack of correctness checks.
• Propose the Tree-of-Thought framework to enable backtracking and expanded solution exploration.
• Demonstrate ToT with a Sudoku solver and assess its performance on Sudoku benchmarks.
• Lay out the architecture, training algorithm, and system components for ToT.
• Discuss limitations and future extensions for generic problem solving with ToT.

Proposed method

• Augment an LLM with a prompter agent, a checker module, a memory module, and a ToT controller to enable tree-like search.
• Use the checker to validate intermediate solutions and permit backtracking via the ToT controller.
• Store conversation history and problem states in memory to guide future prompts and exploration.
• Employ a policy-network based ToT controller (and prompter) trained with a REINFORCE-style method in a multi-agent setup.
• Formalize ToT problem solving as multi-round interactions where the LLM provides short-range reasoning steps guided by agents.

Experimental results

Research questions

• RQ1 Can ToT improve long-range reasoning and solution exploration for complex problems beyond short-range reasoning capability of standard LLMs?
• RQ2 How do the prompter, checker, memory, and controller components interact to enable backtracking and improved problem solving?
• RQ3 Does a ToT-based Sudoku solver achieve higher success rates on benchmark puzzles compared with zero-shot and CoT-based prompts?
• RQ4 What are the limitations of rule-based checkers and controllers, and how might neural components improve performance?

Key findings

• ToT-based Sudoku solver achieved higher success rates than zero-shot and CoT-based solvers on three Sudoku benchmarks (3x3, 4x4, 5x5) as described in the experiments.
• Rule-based ToT controller and checker enable backtracking and memory-assisted exploration of the solution space.
• The ToT framework increases the number of computation steps through multi-round interaction, improving long-range reasoning.
• In the 3x3 puzzle set, the ToT solver solved all puzzles, outperforming other baselines by about 11% in improvement.
• The authors note limitations of the rule-based components and propose neural controller and checker as future enhancements.
• The approach is modular and potentially applicable to broader mathematical and logical reasoning tasks beyond Sudoku.