[论文解读] Improve Mathematical Reasoning in Language Models by Automated Process Supervision

Liangchen Luo, Yinxiao Liu+10 more|arXiv (Cornell University)|Jun 5, 2024

Business Process Modeling and Analysis被引用 6

一句话总结

本文介绍 OmegaPRM，一种将分而治之的蒙特卡洛树搜索方法，用于自动生成用于训练 Process Reward Models 的逐步过程监督数据，从而提升大语言模型在数学推理上的表现，在 MATH 上达到 69.4%。

ABSTRACT

Complex multi-step reasoning tasks, such as solving mathematical problems or generating code, remain a significant hurdle for even the most advanced large language models (LLMs). Verifying LLM outputs with an Outcome Reward Model (ORM) is a standard inference-time technique aimed at enhancing the reasoning performance of LLMs. However, this still proves insufficient for reasoning tasks with a lengthy or multi-hop reasoning chain, where the intermediate outcomes are neither properly rewarded nor penalized. Process supervision addresses this limitation by assigning intermediate rewards during the reasoning process. To date, the methods used to collect process supervision data have relied on either human annotation or per-step Monte Carlo estimation, both prohibitively expensive to scale, thus hindering the broad application of this technique. In response to this challenge, we propose a novel divide-and-conquer style Monte Carlo Tree Search (MCTS) algorithm named extit{OmegaPRM} for the efficient collection of high-quality process supervision data. This algorithm swiftly identifies the first error in the Chain of Thought (CoT) with binary search and balances the positive and negative examples, thereby ensuring both efficiency and quality. As a result, we are able to collect over 1.5 million process supervision annotations to train Process Reward Models (PRMs). This fully automated process supervision alongside the weighted self-consistency algorithm is able to enhance LLMs' math reasoning performances. We improved the success rates of the instruction-tuned Gemini Pro model from 51\% to 69.4\% on MATH500 and from 86.4\% to 93.6\% on GSM8K. Similarly, we boosted the success rates of Gemma2 27B from 42.3\% to 58.2\% on MATH500 and from 74.0\% to 92.2\% on GSM8K. The entire process operates without any human intervention or supervision, making our method both financially and ...

研究动机与目标

• 激发对中间步骤反馈的需求，以提升 LLMs 的多步数学推理能力。
• 提出一种自动化、可扩展的逐步监督数据收集方法（Process Reward Models）。
• 证明自动化过程监督结合加权自一致性可以提升在 MATH 上的数学题解答能力。
• 减少对人工标注和暴力枚举采样来获取过程监督数据的依赖。

提出的方法

• 介绍 Process Supervision，并与 Outcome Reward Models (ORMs) 进行对比。
• 提出 OmegaPRM，一种分而治之的蒙特卡洛树搜索，用以高效定位 Chain-of-Thought 解决方案中的第一个错误。
• 对 rollout 进行二分搜索，结合蒙特卡洛估计来识别首个错误步骤，达到 O(k log M) 的复杂度。
• 构建一个状态-动作树，其节点编码问题、部分解和 rollouts；用逐点软标签或替代损失来训练 PRMs。
• 在超过 1.5 百万自动化过程标注数据上训练 PRMs，以实现 PRM 加权多数投票。

Figure 1 : Example tree structure built with our proposed OmegaPRM algorithm. Each node in the tree indicates a state of partial chain-of-thought solution, with information including accuracy of rollouts and other statistics. Each edge indicates an action, i.e. , a reasoning step, from the last stat

实验结果

研究问题

• RQ1 自动化过程监督数据收集是否能够匹配或超过人工标注的过程数据，用于训练 PRMs？
• RQ2 OmegaPRM 能否高效地在 Chain-of-Thought 中定位首个错误，以提供高质量的 PRM 训练信号？
• RQ3 将 PRMs 与加权自一致性结合，是否能提升在如 MATH 这样的标准基准上的数学推理准确性？
• RQ4 不同的 PRM 训练目标（软标签 vs 硬标签、两两对比）对准确性的影响？
• RQ5 自动化数据质量和步骤划分对 PRM 效能和推理表现有何影响？

主要发现

• OmegaPRM 使得在零人工成本下收集超过 150 万条过程监督标注数据成为可能。
• 在 OmegaPRM 数据上训练的 PRMs 在 Hendrycks MATH 基准上的准确率达到 69.4%，相比基线模型提高了相对 36%。
• PRM 加权多数投票结合加权自一致性在性能上优于使用人工标注或其他自动数据集的基线方法。
• 软标签逐点 PRM 训练在一个小型 PRM 测试集上达到 70.1% 的准确率，胜过硬标签和两两选项。
• 灵活、语义无关的步骤划分产生了与基于规则的方法相似的步骤长度分布，并维持了性能。

(a) Monte Carlo estimation of a prefix solution.