[Paper Review] LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models

Shibo Hao, Yi Gu+10 more|arXiv (Cornell University)|Apr 8, 2024

Natural Language Processing Techniques4 citations

TL;DR

This paper introduces AutoRace, an automated, task-adaptive evaluation framework for LLM-generated reasoning chains using GPT-4 to assess logical correctness beyond final answers. It also presents LLM Reasoners, a unified library that standardizes reasoning algorithms via a search-reward-world model formulation, enabling systematic comparison. Key results show AutoRace detects 70.4% of incorrect chains missed by answer-only evaluation, and RAP with a world model outperforms ToT by 42% on Blocksworld.

ABSTRACT

Generating accurate step-by-step reasoning is essential for Large Language Models (LLMs) to address complex problems and enhance robustness and interpretability. Despite the flux of research on developing advanced reasoning approaches, systematically analyzing the diverse LLMs and reasoning strategies in generating reasoning chains remains a significant challenge. The difficulties stem from the lack of two key elements: (1) an automatic method for evaluating the generated reasoning chains on different tasks, and (2) a unified formalism and implementation of the diverse reasoning approaches for systematic comparison. This paper aims to close the gap: (1) We introduce AutoRace for fully automated reasoning chain evaluation. Existing metrics rely on expensive human annotations or pre-defined LLM prompts not adaptable to different tasks. In contrast, AutoRace automatically creates detailed evaluation criteria tailored for each task, and uses GPT-4 for accurate evaluation following the criteria. (2) We develop LLM Reasoners, a library for standardized modular implementation of existing and new reasoning algorithms, under a unified formulation of the search, reward, and world model components. With the new evaluation and library, (3) we conduct extensive study of different reasoning approaches (e.g., CoT, ToT, RAP). The analysis reveals interesting findings about different factors contributing to reasoning, including the reward-guidance, breadth-vs-depth in search, world model, and prompt formats, etc.

Motivation & Objective

• To address the lack of automated, task-adaptive evaluation for LLM reasoning chains, which currently relies on expensive human annotations or non-adaptable prompts.
• To unify diverse reasoning algorithms—such as CoT, ToT, and RAP—under a common formalism to enable systematic comparison and implementation.
• To analyze critical factors influencing reasoning performance, including reward guidance, search breadth vs. depth, world model usage, and prompt design.
• To evaluate the reasoning capabilities of leading LLMs (e.g., GPT-4, Claude-3, Llama-2) using both answer accuracy and reasoning quality metrics.
• To reveal task-specific failure modes in reasoning chains, such as factual hallucinations in RAP due to overly detailed prompting, especially in commonsense tasks.

Proposed method

• AutoRace automatically generates task-specific evaluation criteria by summarizing errors in LLM-generated reasoning chains using GPT-4, enabling dynamic, adaptive evaluation without human-in-the-loop.
• The evaluation criteria are used to instruct GPT-4 to assess reasoning chains for logical consistency, factual accuracy, and step-by-step coherence on a per-task basis.
• LLM Reasoners formalizes reasoning as a search process over reasoning states, parameterized by a reward function, world model, and search algorithm (e.g., beam search, MCTS).
• The library provides modular, composable components for reward, world model, and search, with standardized APIs and visualizations for reproducibility and extension.
• A unified formulation enables mapping of existing methods (CoT, ToT, RAP) to specific configurations of the three core components.
• The framework supports plug-and-play integration of new reasoning algorithms and enables controlled ablation studies on design factors.

Experimental results

Research questions

• RQ1 How can reasoning chains be evaluated automatically and accurately across diverse tasks without relying on human-annotated references or fixed prompts?
• RQ2 What are the key design factors—such as reward guidance, search strategy, world model, and prompt format—that influence the quality of LLM reasoning?
• RQ3 How do different reasoning algorithms (e.g., CoT, ToT, RAP) compare in terms of reasoning chain correctness and final answer accuracy across tasks?
• RQ4 To what extent do leading LLMs (e.g., GPT-4, Claude-3) differ in reasoning quality beyond final answer accuracy, and what explains these differences?
• RQ5 How does the inclusion of a world model affect reasoning performance, especially in tasks requiring state tracking over long sequences?

Key findings

• AutoRace detects 70.4% of incorrect reasoning chains that are missed by final-answer-only evaluation, demonstrating strong correlation with human evaluation.
• On StrategyQA, 39% of reasoning chains with correct answers contain logical or factual errors, highlighting the need for reasoning-specific evaluation.
• RAP with a world model outperforms ToT by 42% on the Blocksworld task, showing that explicit state tracking significantly improves reasoning in long-horizon planning.
• Prompt format critically affects reasoning quality: RAP’s iterative sub-question prompting increases factual hallucinations (e.g., incorrect numbers like 7,000), especially in commonsense tasks.
• GPT-4 Turbo and Claude-3 Opus lead in reasoning ability across tasks, with GPT-4 showing superior reasoning quality (AutoRace score of 0.91) on StrategyQA despite similar answer accuracy to weaker models.
• The ranking of models by AutoRace score differs significantly from answer-only rankings, revealing that reasoning quality is a distinct and critical capability beyond answer correctness.