[Paper Review] Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models

Pan Lu, Baolin Peng+6 more|arXiv (Cornell University)|Apr 19, 2023

Topic ModelingComputer Science91 citations

TL;DR

Cha me leon is a plug-and-play framework that lets LLMs compose diverse tools (vision models, web search, Python, heuristics) into NL-like programs to tackle multi-modal reasoning tasks, achieving new state-of-the-art on ScienceQA and TabMWP with GPT-4.

ABSTRACT

Large language models (LLMs) have achieved remarkable progress in solving various natural language processing tasks due to emergent reasoning abilities. However, LLMs have inherent limitations as they are incapable of accessing up-to-date information (stored on the Web or in task-specific knowledge bases), using external tools, and performing precise mathematical and logical reasoning. In this paper, we present Chameleon, an AI system that mitigates these limitations by augmenting LLMs with plug-and-play modules for compositional reasoning. Chameleon synthesizes programs by composing various tools (e.g., LLMs, off-the-shelf vision models, web search engines, Python functions, and heuristic-based modules) for accomplishing complex reasoning tasks. At the heart of Chameleon is an LLM-based planner that assembles a sequence of tools to execute to generate the final response. We showcase the effectiveness of Chameleon on two multi-modal knowledge-intensive reasoning tasks: ScienceQA and TabMWP. Chameleon, powered by GPT-4, achieves an 86.54% overall accuracy on ScienceQA, improving the best published few-shot result by 11.37%. On TabMWP, GPT-4-powered Chameleon improves the accuracy by 17.0%, lifting the state of the art to 98.78%. Our analysis also shows that the GPT-4-powered planner exhibits more consistent and rational tool selection via inferring potential constraints from instructions, compared to a ChatGPT-powered planner. The project is available at https://chameleon-llm.github.io.

Motivation & Objective

• Motivate and address the limitations of standard LLMs in accessing up-to-date information, external tools, and precise reasoning.
• Propose a flexible, plug-and-play framework that synthesizes NL-like programs by orchestrating a diverse tool set.
• Demonstrate effectiveness on multi-modal ScienceQA and tabular TabMWP benchmarks and compare planning quality across LLMs.

Proposed method

• Introduce a module inventory of diverse tools (LLMs, vision models, web search, Python, heuristics).
• Use an LLM-based planner to generate natural-language-like programs that sequence modules to solve a query.
• Execute modules sequentially with cached context and update inputs/caches between steps.
• Avoid domain-specific programming languages by generating NL-like plans that are easy to understand and extend.
• Demonstrate that GPT-4 as planner yields more consistent tool selection than ChatGPT.

Figure 1 : Examples from our Cha me leon approach with GPT-4 on ScienceQA [ 32 ] , a multi-modal question answering benchmark in scientific domains. Cha me leon is adaptive to different queries by synthesizing programs to compose various tools and executing them sequentially to get final answers.

Experimental results

Research questions

• RQ1 Can an LLM-based planner synthesize robust NL-like programs that compose heterogeneous tools for real-world reasoning tasks?
• RQ2 Do plug-and-play modules across vision, web, knowledge retrieval, and computation improve performance on multi-modal and tabular reasoning benchmarks?
• RQ3 How does planner quality (GPT-4 vs ChatGPT) influence tool selection, plan validity, and final accuracy?

Key findings

• Cha me leon with GPT-4 achieves 86.54% accuracy on ScienceQA, surpassing the best published few-shot result by 11.37%.
• On TabMWP, GPT-4-powered Cha me leon reaches 98.78% accuracy, improving the state of the art by 17.0%.
• GPT-4-powered planning shows more consistent and rational tool selection than ChatGPT, inferred constraints from instructions improve planning.
• Ablation shows Knowledge Retrieval and domain/tools modules are essential for performance in both ScienceQA and TabMWP.
• Cha me leon generalizes across domains by using a natural-language-like program to orchestrate diverse tools without task-specific training.

Figure 2 : Two examples from our Cha me leon approach with GPT-4 on TabMWP [ 33 ] , a mathematical reasoning benchmark with tabular contexts. Cha me leon demonstrates flexibility and efficiency in adapting to different queries that require various reasoning abilities.