[Paper Review] Decomposed Prompting: A Modular Approach for Solving Complex Tasks

Tushar Khot, Harsh Trivedi+5 more|arXiv (Cornell University)|Oct 5, 2022

Topic ModelingComputer Science91 citations

TL;DR

Decomposed Prompting (DecomP) decomposes complex tasks into simpler sub-tasks handled by modular sub-task prompts and optionally symbolic or external tools, improving few-shot performance across multiple reasoning tasks and enabling flexible reuse and replacement of components.

ABSTRACT

Few-shot prompting is a surprisingly powerful way to use Large Language Models (LLMs) to solve various tasks. However, this approach struggles as the task complexity increases or when the individual reasoning steps of the task themselves are hard to learn, especially when embedded in more complex tasks. To address this, we propose Decomposed Prompting, a new approach to solve complex tasks by decomposing them (via prompting) into simpler sub-tasks that can be delegated to a library of prompting-based LLMs dedicated to these sub-tasks. This modular structure allows each prompt to be optimized for its specific sub-task, further decomposed if necessary, and even easily replaced with more effective prompts, trained models, or symbolic functions if desired. We show that the flexibility and modularity of Decomposed Prompting allows it to outperform prior work on few-shot prompting using GPT3. On symbolic reasoning tasks, we can further decompose sub-tasks that are hard for LLMs into even simpler solvable sub-tasks. When the complexity comes from the input length, we can recursively decompose the task into the same task but with smaller inputs. We also evaluate our approach on textual multi-step reasoning tasks: on long-context multi-hop QA task, we can more effectively teach the sub-tasks via our separate sub-tasks prompts; and on open-domain multi-hop QA, we can incorporate a symbolic information retrieval within our decomposition framework, leading to improved performance on both tasks. Datasets, Code and Prompts available at https://github.com/allenai/DecomP.

Motivation & Objective

• Motivate improved few-shot solving of complex tasks beyond single-prompt reasoning.
• Introduce a modular decomposition framework where a decomposer prompts sub-task handlers to solve sub-problems.
• Enable reuse and replacement of sub-task handlers, including symbolic tools or external systems.
• Demonstrate improved performance across symbolic manipulation, long-context QA, and open-domain multi-hop QA.

Proposed method

• Propose a decomposer LLM that generates a program-like sequence of sub-tasks for a complex query.
• Instantiate each sub-task via a dedicated sub-task handler which can be another prompting program or a symbolic function.
• Use in-context examples to train the decomposer and each sub-task handler independently (modular prompts).
• Allow recursive or hierarchical decomposition to handle very long inputs or nested sub-problems.
• Integrate external tools (e.g., Elasticsearch) as sub-task handlers for retrieval tasks.
• Evaluate with greedy inference over eight datasets spanning symbolic tasks, long-context QA, and open-domain multi-hop QA.

Experimental results

Research questions

• RQ1 Can decomposed prompting improve few-shot performance on complex tasks compared to standard chain-of-thought prompting?
• RQ2 Does modular decomposition enable replacing or upgrading sub-task handlers without changing the whole system?
• RQ3 Can recursive and hierarchical decomposition generalize to longer inputs and more complex reasoning?
• RQ4 What is the impact of incorporating symbolic or retrieval tools as sub-task handlers on open-domain QA?
• RQ5 How does post-processing or error-correcting sub-tasks affect overall accuracy?

Key findings

• DecomP outperforms standard CoT prompting on several tasks and generalizes better to longer inputs and unseen compositions.
• Recursive decomposition enables length generalization for list-reversal tasks beyond standard CoT.
• Incorporating a retrieval sub-task (Elasticsearch) within the DecomP framework improves open-domain multi-hop QA across multiple datasets.
• Post-processing CoT with targeted sub-task handlers yields large improvements on arithmetic word problems (e.g., 17-point gain on MultiArith, 14-point gain on GSM8K).
• DecomP maintains modularity: sub-task prompts can be individually improved or replaced without reworking the entire system, and external tools can be plugged in as needed.