[Paper Review] What Are Tools Anyway? A Survey from the Language Model Perspective

Zhiruo Wang, Zhoujun Cheng+3 more|arXiv (Cornell University)|Mar 18, 2024

Natural Language Processing Techniques4 citations

TL;DR

This survey proposes a unified definition of tools as external program interfaces callable by language models to extend their capabilities, systematically reviews tooling scenarios and methods, and empirically evaluates compute efficiency and performance gains across benchmarks. It identifies ToolFormer as the most efficient method for mathematical reasoning and TroVE as the most efficient tool-making approach, offering practical guidance on when and how to use tools effectively.

ABSTRACT

Language models (LMs) are powerful yet mostly for text generation tasks. Tools have substantially enhanced their performance for tasks that require complex skills. However, many works adopt the term "tool" in different ways, raising the question: What is a tool anyway? Subsequently, where and how do tools help LMs? In this survey, we provide a unified definition of tools as external programs used by LMs, and perform a systematic review of LM tooling scenarios and approaches. Grounded on this review, we empirically study the efficiency of various tooling methods by measuring their required compute and performance gains on various benchmarks, and highlight some challenges and potential future research in the field.

Motivation & Objective

• To establish a unified, broad definition of tools as external program interfaces callable by language models.
• To systematically categorize and review existing tooling scenarios, including perception, action, and computation tools.
• To empirically evaluate the compute-performance trade-offs of various tooling methods across benchmarks.
• To identify which tasks benefit most from tool integration and which methods offer the highest efficiency.
• To propose improved evaluation metrics and highlight gaps in current benchmarks for tool use and tool-making.

Proposed method

• Proposes a formal definition of an LM-used tool as a function interface to an external program, callable by the language model with input arguments.
• Classifies tools into three functional categories: perception (e.g., get_time()), action (e.g., make_post()), and computation (e.g., calculator).
• Reviews tooling paradigms, including prompting, fine-tuning, and tool-making, with a focus on in-context learning and retrieval-augmented generation.
• Empirically evaluates 10 tooling methods across 5 benchmarks using both inference and training compute, measuring performance gains and cost.
• Introduces a cost-efficiency analysis framework comparing methods by compute (tokens/parameters) and performance (accuracy gain).
• Proposes enhanced evaluation metrics and benchmarks with natural use cases and executable tools to improve realism and reproducibility.

Figure 1: Illustration of tools extending and facilitating LM task-solving.

Experimental results

Research questions

• RQ1 What constitutes a tool from the perspective of a language model, and how can it be formally defined across diverse use cases?
• RQ2 Which tasks benefit most from tool integration, and why do some tasks show performance degradation despite tool use?
• RQ3 How do different tooling methods (e.g., prompting, fine-tuning, tool-making) compare in terms of compute efficiency and performance gain?
• RQ4 What are the key trade-offs between training-time and inference-time costs in tooling approaches?
• RQ5 How can evaluation benchmarks and metrics be improved to better reflect real-world tool usage and tool creation?

Key findings

• ToolFormer achieves the highest efficiency on MATH benchmarks, delivering a 30.4% performance gain with only 0.17 MB of compute.
• TroVE is the most efficient tool-making method, improving accuracy by 12.0–21.0 points at just 1.2–1.4K tokens of compute.
• Multilingual translation tasks show a slight performance drop (-0.2 points) despite similar compute costs, indicating limited benefit from tooling in inherently language-native tasks.
• API-Bank and ToolAlpaca outperform ToolFormer in inference-time efficiency, suggesting they are better suited for low-latency deployment.
• CREATOR and CRAFT are significantly less efficient than TroVE, requiring 3.8–6.0 times more compute for only marginal (0.0–5.0%) accuracy improvements.
• Tasks requiring multi-domain reasoning or external knowledge (e.g., tool-using, tool-making) show the highest performance gains, especially on synthesized benchmarks like API-Bank and ToolAlpaca.

Figure 2: The basic tool use paradigm. LM calls check_weather tool by generating text tokens. This call triggers the server to execute the call and return the output sunny , using which the LM replaces the API call tokens in the response to the user.