[Paper Review] From Summary to Action: Enhancing Large Language Models for Complex Tasks with Open World APIs

Yulong Liu, Yunlong Yuan+6 more|arXiv (Cornell University)|Feb 28, 2024

Business Process Modeling and AnalysisBusiness, Management and Accounting3 citations

TL;DR

This paper proposes Sum2Act, a novel reasoning framework that enhances large language models (LLMs) for complex real-world tasks by integrating massive open-world APIs through a two-component pipeline: a router and a state manager. At each step, the LLM summarizes prior results to maintain context and guide decisions, significantly improving performance over ReAct and DFSDT on the ToolBench benchmark with 16,000+ real-world APIs.

ABSTRACT

The distinction between humans and animals lies in the unique ability of humans to use and create tools. Tools empower humans to overcome physiological limitations, fostering the creation of magnificent civilizations. Similarly, enabling foundational models like Large Language Models (LLMs) with the capacity to learn external tool usage may serve as a pivotal step toward realizing artificial general intelligence. Previous studies in this field have predominantly pursued two distinct approaches to augment the tool invocation capabilities of LLMs. The first approach emphasizes the construction of relevant datasets for model fine-tuning. The second approach, in contrast, aims to fully exploit the inherent reasoning abilities of LLMs through in-context learning strategies. In this work, we introduce a novel tool invocation pipeline designed to control massive real-world APIs. This pipeline mirrors the human task-solving process, addressing complicated real-life user queries. At each step, we guide LLMs to summarize the achieved results and determine the next course of action. We term this pipeline `from Summary to action', Sum2Act for short. Empirical evaluations of our Sum2Act pipeline on the ToolBench benchmark show significant performance improvements, outperforming established methods like ReAct and DFSDT. This highlights Sum2Act's effectiveness in enhancing LLMs for complex real-world tasks.

Motivation & Objective

• To address the limitations of existing LLM tool-use frameworks that fail to effectively manage dynamic, real-world API interactions.
• To improve LLM reasoning in complex, open-ended tasks by maintaining accurate task state awareness through continuous summarization of historical outcomes.
• To enable robust handling of API failures and path exploration without error propagation, inspired by human task-solving behavior.
• To integrate both text-based and visual APIs (e.g., SDXL, ControlNet, VQA) for multimodal task execution.
• To develop a scalable, efficient reasoning pipeline that balances breadth of search with contextual fidelity, outperforming tree-based and chain-of-thought methods.

Proposed method

• The Sum2Act pipeline consists of a router and a state manager, where the router selects actions based on the current task state and summarized history.
• At each step, the LLM generates a summary of past actions and outcomes, which is used to inform the next decision, reducing context length and improving focus.
• The state manager validates action results, identifies failure reasons, and enables reflection to correct errors, preventing error propagation.
• The framework supports multi-directional exploration of API paths, avoiding redundant actions and missed information from failed branches.
• It integrates diverse real-world APIs from the ToolBench benchmark, including visual APIs like SDXL, ControlNet, and Blip for image generation and VQA.
• Task decomposition is optionally applied, though empirical results show only marginal gains, indicating that the core framework’s strength lies in its state-aware reasoning rather than structural decomposition.

Experimental results

Research questions

• RQ1 Can continuous summarization of historical outcomes improve LLM reasoning in complex, open-world API environments?
• RQ2 How does Sum2Act’s state management mechanism reduce error propagation compared to CoT and ReAct?
• RQ3 To what extent can Sum2Act explore multiple API paths effectively without redundant or failed action repetition?
• RQ4 Does integrating visual APIs enhance LLM performance on multimodal tasks compared to text-only API use?
• RQ5 How does Sum2Act compare in performance to established methods like ReAct and DFSDT on a large-scale benchmark with 16,000+ real-world APIs?

Key findings

• Sum2Act outperforms ReAct and DFSDT on the ToolBench benchmark, demonstrating superior performance in handling complex, real-world tasks involving 16,000+ APIs across 49 categories.
• The integration of visual APIs such as SDXL, ControlNet, and Blip enables the model to perform advanced image generation, editing, and visual question answering tasks effectively.
• The use of task decomposition provides only marginal improvements in Pass Rate and Win Rate, suggesting that the core reasoning mechanism of Sum2Act is more impactful than structural task breakdown.
• The reflection and failure-analysis capability in the state manager enables the model to identify and correct errors, maintaining accurate task state awareness across complex workflows.
• Sum2Act maintains a manageable context length through summarization while preserving critical information, enabling sustained reasoning over long, multi-step tasks.
• The framework successfully handles dynamic API responses and explores multiple paths without repeating failed actions, demonstrating robustness in open-world environments.