[Paper Review] Automatic Chain of Thought Prompting in Large Language Models

Zhuosheng Zhang, Aston Zhang+2 more|arXiv (Cornell University)|Oct 7, 2022

Topic Modeling233 citations

TL;DR

Auto-CoT automatically constructs diverse chain-of-thought demonstrations using clustering and zero-shot reasoning to achieve competitive or superior performance versus manually crafted demonstrations across ten reasoning benchmarks.

ABSTRACT

Large language models (LLMs) can perform complex reasoning by generating intermediate reasoning steps. Providing these steps for prompting demonstrations is called chain-of-thought (CoT) prompting. CoT prompting has two major paradigms. One leverages a simple prompt like "Let's think step by step" to facilitate step-by-step thinking before answering a question. The other uses a few manual demonstrations one by one, each composed of a question and a reasoning chain that leads to an answer. The superior performance of the second paradigm hinges on the hand-crafting of task-specific demonstrations one by one. We show that such manual efforts may be eliminated by leveraging LLMs with the "Let's think step by step" prompt to generate reasoning chains for demonstrations one by one, i.e., let's think not just step by step, but also one by one. However, these generated chains often come with mistakes. To mitigate the effect of such mistakes, we find that diversity matters for automatically constructing demonstrations. We propose an automatic CoT prompting method: Auto-CoT. It samples questions with diversity and generates reasoning chains to construct demonstrations. On ten public benchmark reasoning tasks with GPT-3, Auto-CoT consistently matches or exceeds the performance of the CoT paradigm that requires manual designs of demonstrations. Code is available at https://github.com/amazon-research/auto-cot

Motivation & Objective

• Motivate reducing manual effort in chain-of-thought prompting by automating demonstration design.
• Show that diversity in automatically generated demonstrations improves CoT performance.
• Demonstrate Auto-CoT’s effectiveness across arithmetic, commonsense, and symbolic reasoning tasks.

Proposed method

• Cluster questions using Sentence-BERT representations and k-means to create diverse groups.
• From each cluster, select a representative question and generate a reasoning chain via Zero-Shot-CoT prompt to form a demonstration.
• Construct k demonstrations (one per cluster) and concatenate them with the test question as in-context prompts.
• Apply simple heuristics to filter demonstrations (e.g., length constraints on questions and rationales).
• Evaluate Auto-CoT with GPT-3 (text-davinci-002) and Codex against Zero-Shot-CoT, Zero-Shot-CoT, Few-Shot, and Manual-CoT baselines.
• Compare performance across ten datasets spanning arithmetic, commonsense, and symbolic reasoning.

Experimental results

Research questions

• RQ1 Can automatic construction of CoT demonstrations, driven by diversity, match or exceed manually designed demonstrations?
• RQ2 Does diversity-based sampling mitigate errors from zero-shot reasoning chains (misleading by similarity) in Auto-CoT?
• RQ3 How does Auto-CoT perform across diverse reasoning tasks and with different LLM backbones (GPT-3 and Codex)?

Key findings

• Auto-CoT consistently matches or exceeds Manual-CoT across ten reasoning datasets when using GPT-3.
• On arithmetic datasets, Auto-CoT achieves near or better accuracy than Manual-CoT (e.g., MultiArith: 92.0 vs 91.7).
• Auto-CoT remains competitive or superior to Manual-CoT on GSM8K, AddSub, AQuA, SingleEq, SVAMP, CSQA, Strategy, Letter, and Coin datasets.
• Using Codex, Auto-CoT remains competitive and often superior to Manual-CoT (e.g., MultiArith: 93.2 vs 96.8 for Manual-CoT, Auto-CoT: 93.2; AddSub: 91.9 vs 84.6 for Manual-CoT).
• Auto-CoT’s diversity-driven sampling demonstrates robustness to wrong demonstrations, outperforming in-cluster sampling as wrong demonstrations increase.
• Auto-CoT adapts to streaming scenarios via Auto-CoT* bootstrapping, showing comparable performance to Manual-CoT after initial batches.