[Paper Review] Large Language Models are Zero-Shot Reasoners

Takeshi Kojima, Shixiang Gu+3 more|arXiv (Cornell University)|May 24, 2022

Topic Modeling1,101 citations

TL;DR

A fixed prompt, “Let’s think step by step,” enables zero-shot chain-of-thought reasoning across diverse tasks, yielding large performance gains over standard zero-shot prompts and approaching few-shot CoT levels on several benchmarks.

ABSTRACT

Pretrained large language models (LLMs) are widely used in many sub-fields of natural language processing (NLP) and generally known as excellent few-shot learners with task-specific exemplars. Notably, chain of thought (CoT) prompting, a recent technique for eliciting complex multi-step reasoning through step-by-step answer examples, achieved the state-of-the-art performances in arithmetics and symbolic reasoning, difficult system-2 tasks that do not follow the standard scaling laws for LLMs. While these successes are often attributed to LLMs' ability for few-shot learning, we show that LLMs are decent zero-shot reasoners by simply adding "Let's think step by step" before each answer. Experimental results demonstrate that our Zero-shot-CoT, using the same single prompt template, significantly outperforms zero-shot LLM performances on diverse benchmark reasoning tasks including arithmetics (MultiArith, GSM8K, AQUA-RAT, SVAMP), symbolic reasoning (Last Letter, Coin Flip), and other logical reasoning tasks (Date Understanding, Tracking Shuffled Objects), without any hand-crafted few-shot examples, e.g. increasing the accuracy on MultiArith from 17.7% to 78.7% and GSM8K from 10.4% to 40.7% with large InstructGPT model (text-davinci-002), as well as similar magnitudes of improvements with another off-the-shelf large model, 540B parameter PaLM. The versatility of this single prompt across very diverse reasoning tasks hints at untapped and understudied fundamental zero-shot capabilities of LLMs, suggesting high-level, multi-task broad cognitive capabilities may be extracted by simple prompting. We hope our work not only serves as the minimal strongest zero-shot baseline for the challenging reasoning benchmarks, but also highlights the importance of carefully exploring and analyzing the enormous zero-shot knowledge hidden inside LLMs before crafting finetuning datasets or few-shot exemplars.

Motivation & Objective

• Motivate investigation into zero-shot reasoning capabilities of large language models (LLMs).
• Evaluate whether a single, task-agnostic prompt can elicit multi-step reasoning without in-task exemplars.
• Quantify performance gains across diverse reasoning benchmarks (arithmetic, symbolic, commonsense, logical).
• Assess model-size effects and generality of the zero-shot prompting approach.

Proposed method

• Introduce Zero-shot-CoT: a single, fixed prompt that induces chain-of-thought reasoning without in-task examples.
• Use a two-stage prompting process: first elicit a reasoning path, then extract the final answer from the reasoning text.
• Apply a simple trigger sentence (e.g., “Let’s think step by step”) before each answer to prompt step-by-step reasoning.
• Employ answer extraction prompts to parse the final answer from the model output.
• Evaluate across 12 datasets spanning arithmetic, commonsense, symbolic, and other logical reasoning tasks, using greedy decoding for determinism.

Experimental results

Research questions

• RQ1 Can a single zero-shot prompt elicit multi-step reasoning across very different reasoning tasks?
• RQ2 How does the zero-shot-CoT prompt affect performance on arithmetic, symbolic, commonsense, and logical benchmarks compared to standard zero-shot prompting?
• RQ3 Does model size influence the effectiveness of zero-shot chain-of-thought prompting?
• RQ4 Is zero-shot-CoT a versatile baseline that generalizes across diverse tasks without per-task prompt engineering?

Key findings

• Zero-shot-CoT substantially outperforms standard zero-shot prompting on many arithmetic and symbolic tasks (e.g., MultiArith and GSM8K).
• Zero-shot-CoT achieves large gains over zero-shot baselines (e.g., MultiArith 17.7% to 78.7%; GSM8K 10.4% to 40.7% with InstructGPT-3).
• Similar magnitudes of improvement are observed with PaLM (540B) in arithmetic benchmarks.
• Few-shot-CoT remains superior when carefully crafted exemplars are available, but Zero-shot-CoT provides a strong, broadly applicable zero-shot baseline.
• Model size amplifies the benefits of chain-of-thought prompting; smaller models show limited gains without CoT, while larger models scale benefits with Zero-shot-CoT.
• Zero-shot-CoT also yields plausible reasoning paths across tasks, even when final answers are incorrect, suggesting broader cognitive capabilities being tapped.