[论文解读] Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspective

Zhiyuan Zeng, Qinyuan Cheng+7 more|arXiv (Cornell University)|Dec 18, 2024

Auction Theory and Applications被引用 4

一句话总结

本文提出了一种基于强化学习的路线图，旨在复现OpenAI o1的专家级推理能力，重点聚焦于四个核心组件：通过人类推理风格进行策略初始化、结合结果与过程信号的奖励设计、如MCTS和Best-of-N等搜索策略，以及通过搜索生成数据进行微调的可扩展学习。其主要贡献是一个系统化、模块化的框架，统一了现有的开源o1复现工作，并通过搜索与学习的结合，为实现类似性能提供了可扩展的路径。

ABSTRACT

OpenAI o1 represents a significant milestone in Artificial Inteiligence, which achieves expert-level performances on many challanging tasks that require strong reasoning ability.OpenAI has claimed that the main techinique behinds o1 is the reinforcement learining. Recent works use alternative approaches like knowledge distillation to imitate o1's reasoning style, but their effectiveness is limited by the capability ceiling of the teacher model. Therefore, this paper analyzes the roadmap to achieving o1 from the perspective of reinforcement learning, focusing on four key components: policy initialization, reward design, search, and learning. Policy initialization enables models to develop human-like reasoning behaviors, equipping them with the ability to effectively explore solution spaces for complex problems. Reward design provides dense and effective signals via reward shaping or reward modeling, which is the guidance for both search and learning. Search plays a crucial role in generating high-quality solutions during both training and testing phases, which can produce better solutions with more computation. Learning utilizes the data generated by search for improving policy, which can achieve the better performance with more parameters and more searched data. Existing open-source projects that attempt to reproduce o1 can be seem as a part or a variant of our roadmap. Collectively, these components underscore how learning and search drive o1's advancement, making meaningful contributions to the development of LLM.

研究动机与目标

• 识别并系统化复现OpenAI o1专家级推理能力的关键组件。
• 提供一个全面、模块化的框架，利用强化学习原理复现o1。
• 阐明现有开源o1复现项目如何与该路线图对齐或偏离。
• 通过系统性整合搜索与学习，解决大语言模型推理能力的可扩展性挑战。

提出的方法

• 通过语言理解、世界知识和基础推理的预训练进行策略初始化，随后通过指令微调引入人类推理行为，如目标澄清和任务分解。
• 采用基于结果和基于过程的双重奖励设计，方法包括奖励塑形、学习的奖励模型以及集成方法，以提升信号密度和泛化能力。
• 基于内部与外部引导（如自我评估、价值函数和启发式规则）的搜索策略，结合MCTS、Best-of-N和束搜索等规划算法，生成高质量的解题候选。
• 通过策略梯度方法（如PPO、DPO）和行为克隆进行学习，利用搜索生成的数据对策略进行微调，以迭代提升性能。
• 引入世界模型以实现在模拟环境中的规划与搜索，减少对实时环境交互的依赖。
• 使用连续表示将多模态信息（如图像）整合到思维链推理中，降低延迟并提升对齐性。

Figure 1: The overview of this roadmap including policy initialization, reward design, search and learning.

实验结果

研究问题

• RQ1 使用强化学习复现o1推理能力所需的关键组件是什么？
• RQ2 如何有效通过策略初始化在大语言模型中引入人类推理行为？
• RQ3 哪些奖励设计策略能在复杂推理任务中最大化信号密度与泛化能力？
• RQ4 在计算资源扩展下，MCTS和Best-of-N等搜索策略如何提升解的质量？
• RQ5 世界模型在实现可扩展、安全且可逆的规划方面发挥什么作用，以支持真实世界智能体的部署？

主要发现

• 现有开源o1复现项目（如Marco-o1和o1-coder）被证明是本路线图的具体实现，验证了其通用性。
• 在拒绝采样或MCTS生成数据上应用DPO和SFT等强化学习方法可显著提升推理性能，结合蒸馏后性能进一步提升。
• MCTS过程中引入自我反思可提高搜索效率与解的质量，证明了内部反馈在规划中的价值。
• 基于结果的奖励模型在具有标准答案的推理任务中表现有效，而基于过程和学习的奖励模型对非推理或对齐任务至关重要。
• 世界模型通过允许智能体在真实环境交互前模拟交互过程，实现安全、可逆的规划，对真实世界部署至关重要。
• 将多模态数据（如图像）整合到思维链推理中会增加推理长度与延迟，提示需要采用连续紧凑的表示形式。

Figure 3: The visualization of the interaction between agent and einvironment in reinforcement learning for LLMs. Left: traditional reinforcement learning. Right: reinforcement learning for LLMs. The figure only visualizes the step-level action for simplicity. In fact, the action of LLM can be eithe