[Paper Review] A Comparative Study on Reasoning Patterns of OpenAI's o1 Model

Siwei Wu, Zhongyuan Peng+15 more|arXiv (Cornell University)|Oct 17, 2024

Business Process Modeling and Analysis10 citations

TL;DR

The paper compares OpenAI's o1 model with several test-time compute methods across math, coding, and commonsense tasks, analyzes six o1 reasoning patterns, and releases code and data.

ABSTRACT

Enabling Large Language Models (LLMs) to handle a wider range of complex tasks (e.g., coding, math) has drawn great attention from many researchers. As LLMs continue to evolve, merely increasing the number of model parameters yields diminishing performance improvements and heavy computational costs. Recently, OpenAI's o1 model has shown that inference strategies (i.e., Test-time Compute methods) can also significantly enhance the reasoning capabilities of LLMs. However, the mechanisms behind these methods are still unexplored. In our work, to investigate the reasoning patterns of o1, we compare o1 with existing Test-time Compute methods (BoN, Step-wise BoN, Agent Workflow, and Self-Refine) by using OpenAI's GPT-4o as a backbone on general reasoning benchmarks in three domains (i.e., math, coding, commonsense reasoning). Specifically, first, our experiments show that the o1 model has achieved the best performance on most datasets. Second, as for the methods of searching diverse responses (e.g., BoN), we find the reward models' capability and the search space both limit the upper boundary of these methods. Third, as for the methods that break the problem into many sub-problems, the Agent Workflow has achieved better performance than Step-wise BoN due to the domain-specific system prompt for planning better reasoning processes. Fourth, it is worth mentioning that we have summarized six reasoning patterns of o1, and provided a detailed analysis on several reasoning benchmarks.

Motivation & Objective

• Motivate understanding of how inference-time strategies affect LLM reasoning performance.
• Evaluate o1 against traditional test-time compute methods on diverse reasoning benchmarks.
• Characterize the reasoning patterns (types) displayed by o1 across tasks.
• Provide insights to guide development of future foundation models and inference strategies.

Proposed method

• Use GPT-4o as backbone to compare o1 with BoN, Step-wise BoN, Agent Workflow, and Self-Refine on four benchmarks.
• Employ four baselines and four test-time compute methods on HotpotQA, Collie, USACO, and AIME.
• Filter commonsense datasets using a voting approach among multiple LLMs (Llama3, Qwen, Claude, Yi).
• Analyze six o1 reasoning patterns: Systematic Analysis, Method Reuse, Divide and Conquer, Self-Refinement, Context Identification, Emphasizing Constraints.
• Compute and compare accuracy/score across tasks and examine token usage in Step-wise BoN.
• Release code and dataset at the referenced GitHub repository.

Experimental results

Research questions

• RQ1 How does o1's performance compare to BoN, Step-wise BoN, Agent Workflow, and Self-Refine across math, coding, and commonsense tasks?
• RQ2 What inference-time strategies maximize o1's advantages, and what are their limitations?
• RQ3 What are the six reasoning patterns exhibited by o1, and how do they vary by task?
• RQ4 How does token usage for reasoning differ across tasks and strategies?
• RQ5 What factors bound the effectiveness of reward models and search spaces in BoN-style methods?

Key findings

• OpenAI’s o1 generally achieves the best results across benchmarks, especially in math and coding tasks.
• Agent Workflow significantly improves performance across all benchmarks, approaching o1 levels, while BoN, Step-wise BoN, and Self-Refine show more limited gains.
• Six o1 reasoning patterns identified: Systematic Analysis, Method Reuse, Divide and Conquer, Self-Refinement, Context Identification, Emphasizing Constraints; DC and SR are most prominent overall.
• Step-wise BoN produces long reasoning chains and many tokens, and its effectiveness depends on task demands and context length.
• Reward models and search space size substantially constrain BoN-type methods, with human reward showing notable improvements on HotpotQA.
• Reasoning token length varies by task and is not simply correlated with input prompt length; harder tasks (coding/math) tend to need more reasoning tokens.