[Paper Review] Least-to-Most Prompting Enables Complex Reasoning in Large Language Models

Denny Zhou, Nathanael Schärli+9 more|arXiv (Cornell University)|May 21, 2022

Topic Modeling317 citations

TL;DR

Least-to-most prompting decomposes hard problems into a sequence of easier subproblems and solves them in order, enabling LLMs to generalize to harder tasks without training. It outperforms standard and chain-of-thought prompting on symbolic manipulation, compositional generalization, and math reasoning benchmarks.

ABSTRACT

Chain-of-thought prompting has demonstrated remarkable performance on various natural language reasoning tasks. However, it tends to perform poorly on tasks which requires solving problems harder than the exemplars shown in the prompts. To overcome this challenge of easy-to-hard generalization, we propose a novel prompting strategy, least-to-most prompting. The key idea in this strategy is to break down a complex problem into a series of simpler subproblems and then solve them in sequence. Solving each subproblem is facilitated by the answers to previously solved subproblems. Our experimental results on tasks related to symbolic manipulation, compositional generalization, and math reasoning reveal that least-to-most prompting is capable of generalizing to more difficult problems than those seen in the prompts. A notable finding is that when the GPT-3 code-davinci-002 model is used with least-to-most prompting, it can solve the compositional generalization benchmark SCAN in any split (including length split) with an accuracy of at least 99% using just 14 exemplars, compared to only 16% accuracy with chain-of-thought prompting. This is particularly noteworthy because neural-symbolic models in the literature that specialize in solving SCAN are trained on the entire training set containing over 15,000 examples. We have included prompts for all the tasks in the Appendix.

Motivation & Objective

• Motivate easy-to-hard generalization in large language models and address limits of chain-of-thought prompting.
• Introduce a two-stage prompting framework that decomposes problems and then solves subproblems sequentially.
• Show that least-to-most prompting enables generalization to harder problems across symbolic manipulation, SCAN compositional generalization, and math reasoning datasets.

Proposed method

• Two-stage prompting: (i) decomposition prompts demonstrate breaking a problem into subproblems; (ii) subproblem solving prompts show sequential resolution using prior answers.
• Prompts are few-shot; no model training or finetuning is required.
• Prompts can be combined with or used independently of chain-of-thought or self-consistency decoding.
• Prompts are designed to teach the model to build solutions using outputs of previously solved subproblems (base case and recursive step).
• Evaluation across multiple tasks includes symbolic manipulation (last-letter concatenation), SCAN for compositional generalization, and math reasoning (GSM8K and DROP).

Experimental results

Research questions

• RQ1 Can least-to-most prompting enable LLMs to solve problems that are harder than those seen in prompts?
• RQ2 Does decomposing problems into subproblems improve generalization on symbolic, compositional, and mathematical reasoning tasks?
• RQ3 How does least-to-most prompting compare to chain-of-thought prompting across these domains?
• RQ4 To what extent does prompting alone (without training) achieve high accuracy on benchmarks like SCAN and GSM8K/DROP?

Key findings

• On last-letter concatenation, least-to-most prompting achieves higher accuracy than chain-of-thought prompting, especially as list length increases.
• On SCAN, code-davinci-002 with least-to-most prompting reaches 99.7% accuracy under length split, outperforming standard and chain-of-thought prompting by large margins.
• On GSM8K and DROP, least-to-most prompting outperforms chain-of-thought prompting, with notable gains on DROP and comparable gains in GSM8K, particularly for problems requiring many steps.
• Decomposition-based prompts enable solving problems longer or more complex than those demonstrated, without model training.
• Error analysis shows most failures are due to concatenation/step interpretation rather than incorrect subproblem solutions.