[Paper Review] PAL: Program-aided Language Models

Luyu Gao, Aman Madaan+6 more|arXiv (Cornell University)|Nov 18, 2022

Topic Modeling104 citations

TL;DR

PaL uses a large language model to generate interleaved natural language and Python code as intermediate reasoning steps, then offloads solving to a Python interpreter to produce final answers. It achieves state-of-the-art few-shot accuracy across 13 math, symbolic, and algorithmic tasks, often outperforming much larger models using chain-of-thought prompting.

ABSTRACT

Large language models (LLMs) have recently demonstrated an impressive ability to perform arithmetic and symbolic reasoning tasks, when provided with a few examples at test time ("few-shot prompting"). Much of this success can be attributed to prompting methods such as "chain-of-thought'', which employ LLMs for both understanding the problem description by decomposing it into steps, as well as solving each step of the problem. While LLMs seem to be adept at this sort of step-by-step decomposition, LLMs often make logical and arithmetic mistakes in the solution part, even when the problem is decomposed correctly. In this paper, we present Program-Aided Language models (PAL): a novel approach that uses the LLM to read natural language problems and generate programs as the intermediate reasoning steps, but offloads the solution step to a runtime such as a Python interpreter. With PAL, decomposing the natural language problem into runnable steps remains the only learning task for the LLM, while solving is delegated to the interpreter. We demonstrate this synergy between a neural LLM and a symbolic interpreter across 13 mathematical, symbolic, and algorithmic reasoning tasks from BIG-Bench Hard and other benchmarks. In all these natural language reasoning tasks, generating code using an LLM and reasoning using a Python interpreter leads to more accurate results than much larger models. For example, PAL using Codex achieves state-of-the-art few-shot accuracy on the GSM8K benchmark of math word problems, surpassing PaLM-540B which uses chain-of-thought by absolute 15% top-1. Our code and data are publicly available at http://reasonwithpal.com/ .

Motivation & Objective

• Motivate robust reasoning with LLMs by separating problem decomposition from solution execution.
• Demonstrate that generating executable code as reasoning improves accuracy over chain-of-thought in diverse benchmarks.
• Show that PaL yields state-of-the-art few-shot performance on gsm8k and competitive gains on symbolic/algorithmic tasks.

Proposed method

• LLM generates interleaved natural language and programming language steps as intermediate reasoning (t = NL ∪ PL).
• The final answer is produced by executing the generated code in a Python interpreter; the LLM does not output the final answer in the prompt.
• Prompts are crafted with meaningful variable names and optional NL comments; execution results can be fed back if needed (post-hoc execution used in experiments).
• Experiments compare Direct prompting, chain-of-thought prompting, and PaL across math, symbolic, and algorithmic tasks, using Codex as the base LLM.
• Evaluation includes fixed-prompt few-shot settings and, for gsm8k, majority-vote sampling (k>1) to boost accuracy.

Experimental results

Research questions

• RQ1 Can program-generated reasoning steps combined with an external interpreter outperform chain-of-thought prompting across math, symbolic, and algorithmic tasks?
• RQ2 How does PaL perform on large-number arithmetic and robustness to prompt variations (variable names, NL comments) across diverse benchmarks?
• RQ3 Is the benefit of PaL due to code-generation prompts, the interpreter, or their combination?
• RQ4 How does PaL compare to larger LLMs trained with chain-of-thought on standard benchmarks like gsm8k?

Key findings

• PaL with Codex achieves state-of-the-art few-shot accuracy on gsm8k, outperforming larger models that use chain-of-thought by a notable margin.
• On gsm-hard (large numbers), PaL remains robust while direct and CoT prompts degrade significantly.
• Across symbolic reasoning and algorithmic tasks from BIG-Bench Hard, PaL substantially outperforms CoT and closes gaps to solving tasks with high accuracy (e.g., colored objects, penguins, dates, object counting, repeated copy).
• PaL improvements persist across weaker base LMs and with LMs primarily trained for natural language if they have adequate code modeling ability.
• Ablations show meaningful variable names and NL comments contribute to PaL’s effectiveness; removing them reduces performance.