[Paper Review] EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instruction

Siyu Yuan, Kaitao Song+6 more|arXiv (Cornell University)|Jan 11, 2024

Topic Modeling4 citations

TL;DR

EASYTOOL proposes a framework that transforms diverse, lengthy, and inconsistent tool documentation into concise, unified, and standardized tool instructions to improve LLM-based agents' tool utilization. By purifying essential information and generating structured, example-rich tool descriptions, EASYTOOL reduces token consumption and significantly boosts performance across multiple benchmarks, including a 28.3% absolute accuracy gain on FuncQA with ChatGPT using ReAct prompting.

ABSTRACT

To address intricate real-world tasks, there has been a rising interest in tool utilization in applications of large language models (LLMs). To develop LLM-based agents, it usually requires LLMs to understand many tool functions from different tool documentation. But these documentations could be diverse, redundant or incomplete, which immensely affects the capability of LLMs in using tools. To solve this, we introduce EASYTOOL, a framework transforming diverse and lengthy tool documentation into a unified and concise tool instruction for easier tool usage. EasyTool purifies essential information from extensive tool documentation of different sources, and elaborates a unified interface (i.e., tool instruction) to offer standardized tool descriptions and functionalities for LLM-based agents. Extensive experiments on multiple different tasks demonstrate that EasyTool can significantly reduce token consumption and improve the performance of tool utilization in real-world scenarios. Our code will be available at \url{https://github.com/microsoft/JARVIS/} in the future.

Motivation & Objective

• To address the challenge of inconsistent, redundant, and incomplete tool documentation that hinders LLM-based agents in selecting and using tools effectively.
• To reduce the token burden of tool documentation in LLM inference by generating compact, standardized tool instructions.
• To improve the performance of LLM-based agents in real-world tool usage tasks without requiring fine-tuning or access to model weights.
• To enable plug-and-play integration of external tools into LLM agents through a unified, human- and model-readable tool instruction format.

Proposed method

• EASYTOOL extracts and consolidates essential information from heterogeneous tool documentation sources (e.g., RapidAPI, Hugging Face, HFModel) into a single, unified tool instruction.
• It applies a multi-step refinement pipeline: information purification, functional abstraction, and structured formatting using a consistent template with name, description, parameters, and example usage.
• The framework generates concise, example-driven tool instructions that include scenario-based usage guidance and parameter specifications.
• It leverages LLMs to synthesize tool instructions from raw documentation, ensuring clarity, completeness, and consistency across tools.
• The method supports both open-source and black-box LLMs by avoiding model fine-tuning and instead relying on prompt engineering with refined tool descriptions.
• It is designed to be extensible and scalable, applicable to tools with minimal or no documentation, as demonstrated on FuncQA.

Experimental results

Research questions

• RQ1 Can a unified, concise tool instruction format significantly improve LLM agents’ ability to select and use external tools in real-world tasks?
• RQ2 How does EASYTOOL reduce token consumption while maintaining or improving tool utilization performance?
• RQ3 To what extent can EASYTOOL enhance LLM agents on tasks with incomplete or minimal tool documentation?
• RQ4 Does the use of standardized, example-rich tool instructions outperform few-shot demonstrations and chain-of-thought prompting in complex reasoning tasks?
• RQ5 Can EASYTOOL be effectively applied to both open-source and proprietary LLMs without requiring fine-tuning?

Key findings

• EASYTOOL improved the accuracy of Vicuna-30B on the one-hop FuncQA benchmark from 45.00% (with ReAct) to 65.00%, representing a 20.00 percentage point improvement.
• For multi-hop FuncQA questions, the accuracy of Vicuna-30B increased from 7.35% to 11.76% with EASYTOOL, a 4.41 percentage point gain.
• ChatGPT’s accuracy on multi-hop FuncQA rose from 41.17% (with ReAct) to 48.53% when using EASYTOOL, a 7.36 percentage point improvement.
• The tool error rate for ChatGPT dropped from 9.38% to 2.34% on FuncQA with EASYTOOL, indicating better correctness in tool selection and parameter usage.
• EASYTOOL reduced the number of tokens required for tool documentation by up to 70% compared to raw documentation, significantly lowering inference cost.
• The framework demonstrated consistent performance gains across diverse benchmarks, including RestBench and FuncQA, across both open-source and proprietary LLMs.