[Paper Review] Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks

Wenhu Chen, Xueguang Ma+2 more|arXiv (Cornell University)|Nov 22, 2022

Topic Modeling110 citations

TL;DR

PoT prompts LLMs to generate Python-like programs that are executed externally to perform computation, decoupling computation from reasoning, and yielding significant gains over Chain-of-Thought on numerical reasoning benchmarks.

ABSTRACT

Recently, there has been significant progress in teaching language models to perform step-by-step reasoning to solve complex numerical reasoning tasks. Chain-of-thoughts prompting (CoT) is by far the state-of-art method for these tasks. CoT uses language models to perform both reasoning and computation in the multi-step `thought' process. To disentangle computation from reasoning, we propose `Program of Thoughts' (PoT), which uses language models (mainly Codex) to express the reasoning process as a program. The computation is relegated to an external computer, which executes the generated programs to derive the answer. We evaluate PoT on five math word problem datasets (GSM, AQuA, SVAMP, TabMWP, MultiArith) and three financial-QA datasets (FinQA, ConvFinQA, TATQA) for both few-shot and zero-shot setups. Under both few-shot and zero-shot settings, PoT can show an average performance gain over CoT by around 12\% across all the evaluated datasets. By combining PoT with self-consistency decoding, we can achieve SoTA performance on all math problem datasets and near-SoTA performance on financial datasets. All of our data and code are released in Github https://github.com/wenhuchen/Program-of-Thoughts

Motivation & Objective

• Motivate the need to disentangle computation from reasoning in numerical reasoning tasks.
• Propose Program of Thoughts (PoT) that delegates computation to an external interpreter while reasoning in code.
• Evaluate PoT on a broad set of math word problems and financial QA datasets to assess gains over chain-of-thought prompting.
• Investigate zero-shot and few-shot PoT, along with self-consistency decoding, and perform ablation studies to understand components.

Proposed method

• Introduce PoT where the thought process is expressed as Python code rather than natural language rationales.
• Use an external Python interpreter (SymPy) to execute generated programs and obtain answers.
• Prompt LLMs (primarily Codex) with few-shot exemplars or in zero-shot mode to produce programs.
• Optionally combine PoT with self-consistency decoding (PoT+SC) to boost robustness.
• In zero-shot PoT, suppress hash-prefixed comments to discourage reasoning in code comments.
• Linearize heterogeneous inputs (text, tables, conversations) into prompts for the LLM.
• Evaluate on nine datasets spanning Math Word Problems and finance QA (GSM8K, AQuA, SVAMP, TabMWP, MultiArith, FinQA, ConvFinQA, TATQA).

Experimental results

Research questions

• RQ1 Does Program of Thoughts (PoT) improve numerical reasoning accuracy over Chain-of-Thought (CoT) prompting across diverse datasets?
• RQ2 What are the effects of few-shot versus zero-shot PoT, and how does self-consistency decoding interact with PoT?
• RQ3 How do backend models and prompt designs influence PoT performance and stability?
• RQ4 What is the impact of semantic binding and multi-step reasoning in PoT compared to direct equation generation?
• RQ5 Can PoT be effectively integrated with CoT for multi-stage reasoning tasks?

Key findings

• PoT yields average gains of about 12% over CoT across evaluated datasets in zero-shot and few-shot settings.
• PoT+SC (PoT with self-consistency) achieves state-of-the-art-like results on math datasets and approaches best results on financial datasets (excluding GPT-4).
• Zero-shot PoT outperforms zero-shot CoT by an average of 12% on the math datasets; on TabMWP it can exceed few-shot CoT.
• PoT outperforms PaL on several benchmarks (e.g., SVAMP, ASDIV) by notable margins.
• Semantic binding and multi-step reasoning are crucial for PoT’s advantage; removing binding or attempting direct equation generation degrades performance, especially on complex problems.
• Error analysis shows grounding errors in values account for a substantial portion of failures, with logic-generation errors also present; PoT generally reduces arithmetic miscalculations by delegating to a precise interpreter.