[Paper Review] AlphaMath Almost Zero: Process Supervision without Process

Chen Guoxin, Minpeng Liao+2 more|arXiv (Cornell University)|May 6, 2024

Business Process Modeling and Analysis6 citations

TL;DR

AlphaMath uses Monte Carlo Tree Search integrated with a pre-trained LLM and a lightweight value model to autonomously generate high-quality mathematical reasoning without human or GPT-4 annotated process data, enabling step-level evaluation and efficient inference via step-level beam search.

ABSTRACT

Although recent advancements in large language models (LLMs) have significantly improved their performance on various tasks, they still face challenges with complex and symbolic multi-step reasoning, particularly in mathematical reasoning. To bolster the mathematical reasoning capabilities of LLMs, most existing efforts concentrate on seeking assistance from either domain experts or GPT-4 for high-quality process-supervised data, which is not only expensive but also labor-intensive. In our study, we propose an innovative framework, AlphaMath, that bypasses the need for process annotations (from humans or GPTs) by leveraging Monte Carlo Tree Search (MCTS). This framework focuses on unleashing the potential of a well-pretrained LLM to autonomously enhance its mathematical reasoning. Specifically, we integrate a value model with the LLM, automatically generating both process supervision and step-level evaluation signals in MCTS. Furthermore, we propose an efficient inference strategy, step-level beam search, where the value model is crafted to assist the policy model (i.e., LLM) in navigating more effective reasoning paths, rather than solely relying on prior probabilities. The experimental results on both in-domain and out-of-domain datasets demonstrate that even without GPT-4 or human-annotated process supervision, our AlphaMath framework achieves comparable or superior results to previous state-of-the-art methods.

Motivation & Objective

• Motivate reducing annotation costs for mathematical reasoning in LLMs by leveraging the model's intrinsic knowledge.
• Develop a self-evolution framework that uses MCTS to generate and evaluate intermediate reasoning steps without external solutions.
• Introduce a step-level beam search to make inference efficient while guiding the LLM with a value model.
• Demonstrate that AlphaMath can achieve state-of-the-art-like performance on in-domain and out-of-domain math datasets without GPT-4 or human annotations.

Proposed method

• Integrate a pre-trained LLM with a Monte Carlo Tree Search (MCTS) framework to generate and evaluate reasoning steps.
• Introduce a step-level value model Vϕ that estimates the quality of partial solutions and guides the LLM during search.
• Define a reward signal that assigns 0 to non-terminal steps and ±1 to terminal correct/incorrect answers, and train Vϕ via regression to eV(s).
• Use a hybrid evaluation during MCTS with ˆV(st) combining the value model and empirical rollout rewards, controlled by λ.
• Propose step-level beam search (SBS) to replace full MCTS during inference, reducing latency while leveraging the value model to select promising steps.
• Employ an iterative training loop where policy πθ and value Vϕ are updated from MCTS-generated correct/incorrect solution paths, using a multi-task loss that combines next-token probability and value errors.

Experimental results

Research questions

• RQ1 Can a well-pretrained LLM generate high-quality mathematical reasoning without human or GPT-4 annotated solutions when guided by MCTS?
• RQ2 Does a lightweight value model integrated into the LLM improve step-level reasoning and overall problem-solving performance?
• RQ3 Is step-level beam search an effective and efficient alternative to full MCTS for production deployment?
• RQ4 How does AlphaMath perform on in-domain and out-of-domain math datasets without external annotated processes?
• RQ5 Can AlphaMath enhance both domain-specific and general-purpose LLMs, including SFT models, in math reasoning tasks?

Key findings

• AlphaMath achieves competitive or superior results compared to state-of-the-art methods on in-domain datasets (GSM8K, MATH) and out-of-domain datasets (GaoKao2023, OCWCourses, GK2023) without GPT-4 or human-annotated solutions.
• Incorporating a step-level value model with step-level beam search substantially improves reasoning performance over greedy decoding and plain MCTS, with performance increasing as beam size grows.
• Iterative training with MCTS-guided self-evolution improves solution quality across rounds, indicating that self-generated data quality improves with more rounds.
• The approach benefits both domain-specific models (e.g., DeepSeekMath-Base-7B) and general-purpose/SFT models (e.g., Llama3, MARIO), demonstrating broad applicability.
• Step-level beam search offers a favorable trade-off between performance and computation, enabling near-MCTS-level reasoning at reduced latency.