[Paper Review] Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs

Xuan Zhang, Chao Du+4 more|arXiv (Cornell University)|Jun 13, 2024

Business Process Modeling and Analysis6 citations

TL;DR

CPO fine-tunes LLMs using per-step preferred/dispreferred thoughts from Tree-of-Thought search to improve Chain-of-Thought reasoning with much lower inference latency than ToT.

ABSTRACT

The recent development of chain-of-thought (CoT) decoding has enabled large language models (LLMs) to generate explicit logical reasoning paths for complex problem-solving. However, research indicates that these paths are not always deliberate and optimal. The tree-of-thought (ToT) method employs tree-searching to extensively explore the reasoning space and find better reasoning paths that CoT decoding might overlook. This deliberation, however, comes at the cost of significantly increased inference complexity. In this work, we demonstrate that fine-tuning LLMs leveraging the search tree constructed by ToT allows CoT to achieve similar or better performance, thereby avoiding the substantial inference burden. This is achieved through Chain of Preference Optimization (CPO), where LLMs are fine-tuned to align each step of the CoT reasoning paths with those of ToT using the inherent preference information in the tree-search process. Extensive experimental results show that CPO significantly improves LLM performance in solving a variety of complex problems, including question answering, fact verification, and arithmetic reasoning, demonstrating its effectiveness. Our code is available at https://github.com/sail-sg/CPO.

Motivation & Objective

• Motivate improving chain-of-thought reasoning by incorporating ToT-derived preferences beyond the final path.
• Leverage the tree-search process to extract per-step preference information for supervision.
• Develop a training objective that directly optimizes preferred vs. dispreferred thoughts at each step.
• Demonstrate that CPO yields notable gains across QA, fact verification, and arithmetic reasoning without extra inference latency.

Proposed method

• Construct a chain of preference thoughts from the ToT search tree by generating multiple thoughts at each step.
• Evaluate each thought with the LLM to assign a progress score and select the top n thoughts via BFS pruning.
• Create per-step preference data by pairing preferred thoughts with their dispreferred siblings at each step.
• Train the LLM with the Direct Preference Optimization (DPO) objective to maximize the likelihood ratio of preferred vs. dispreferred thoughts.
• Compare CPO against CoT, ToT, and TS-SFT across multiple datasets and base models using LoRA-fine-tuned LLMs.

Experimental results

Research questions

• RQ1 Does per-step preference supervision provide advantages over full-path supervision for improving chain-of-thought reasoning?
• RQ2 Can CPO achieve ToT-like benefits with substantially lower inference latency?
• RQ3 Is CPO robust across different reasoning tasks (QA, fact verification, arithmetic) and model sizes?

Key findings

• CPO yields an average accuracy improvement of 4.3% across tasks and models, with a maximum of 9.7%.
• CPO achieves comparable or better performance than ToT while providing much lower inference latency (57.5x faster on average).
• CPO outperforms TS-SFT on average by 2.7% and up to 10.3% in some cases, due to exploiting both preferred and dispreferred thoughts.
• Per-step preference data is beneficial; increasing the proportion of dispreferred data during training consistently improves performance.