[Paper Review] ToolNet: Connecting Large Language Models with Massive Tools via Tool Graph

Xukun Liu, Zhiyuan Peng+5 more|arXiv (Cornell University)|Feb 29, 2024

Semantic Web and Ontologies4 citations

TL;DR

ToolNet proposes a plug-and-play framework that organizes thousands of tools into a directed graph with weighted transitions, enabling large language models (LLMs) to efficiently navigate and select tools step-by-step. By dynamically updating transition weights based on tool performance, ToolNet improves reasoning efficiency and robustness, achieving up to 2.6x higher token efficiency and superior performance on multi-hop tool learning benchmarks compared to in-context learning baselines.

ABSTRACT

While achieving remarkable progress in a broad range of tasks, large language models (LLMs) remain significantly limited in properly using massive external tools. Existing in-context learning approaches simply format tools into a list of plain text descriptions and input them to LLMs, from which, LLMs generate a sequence of tool calls to solve problems step by step. Such a paradigm ignores the intrinsic dependency between tools and offloads all reasoning loads to LLMs, making them restricted to a limited number of specifically designed tools. It thus remains challenging for LLMs to operate on a library of massive tools, casting a great limitation when confronted with real-world scenarios. This paper proposes ToolNet, a plug-and-play framework that scales up the number of tools to thousands with a moderate increase in token consumption. ToolNet organizes tools into a directed graph. Each node represents a tool, and weighted edges denote tool transition. Starting from an initial tool node, an LLM navigates in the graph by iteratively choosing the next one from its successors until the task is resolved. Extensive experiments show that ToolNet can achieve impressive results in challenging multi-hop tool learning datasets and is resilient to tool failures.

Motivation & Objective

• To address the scalability limitations of existing in-context learning methods that struggle with massive tool libraries due to high token consumption and poor tool selection.
• To enable LLMs to efficiently navigate and reason over thousands of tools by modeling tool dependencies through a directed graph structure.
• To improve tool selection accuracy and resilience to low-quality tools by incorporating feedback-driven, adaptive transition weights in the tool graph.
• To reduce reliance on costly fine-tuning while maintaining strong generalization across diverse, evolving tool repositories.
• To enable rapid integration of new tools and adaptation to updated tools through a lightweight, plug-and-play mechanism.

Proposed method

• ToolNet constructs a directed tool graph where each node represents a tool and weighted edges represent potential transitions between tools based on observed usage patterns.
• The framework uses in-context learning to guide the LLM in selecting successor tools from the current node’s outgoing edges, reducing token consumption by avoiding full tool list prompting.
• An evaluator module probes each tool call, scores its effectiveness, and updates the transition weights in the graph to favor high-performing tools and suppress low-quality ones.
• Transition weights are encoded in a structured format (e.g., integers or scaled values) to improve LLM sensitivity to performance differences, with optimal performance observed at ×10 scaling.
• The system supports dynamic adaptation: as new tools are added or existing tools are updated, the graph can be incrementally modified without retraining the LLM.
• The method is evaluated on APIBank and ToolBench, demonstrating robustness to tool failures and scalability to thousands of tools.

Experimental results

Research questions

• RQ1 How can LLMs be enabled to efficiently navigate and select from a massive library of tools without incurring prohibitive token costs?
• RQ2 Can tool transition patterns be modeled as a sparse, directed graph to improve reasoning efficiency and scalability?
• RQ3 How can LLMs adapt their tool selection strategy in response to feedback on tool performance?
• RQ4 What impact do different weight representation formats (e.g., integers vs. decimals) have on LLM performance and token efficiency?
• RQ5 Can a plug-and-play framework dynamically integrate new tools and adjust to evolving tool repositories without retraining?

Key findings

• ToolNet achieves up to 2.6x higher token efficiency compared to standard in-context learning baselines on multi-hop tool learning benchmarks.
• Performance saturates at a transition weight scaling of ×10, indicating optimal sensitivity to performance differences at this level.
• Removing or scaling down weights to decimals significantly degrades performance, highlighting the importance of proper weight formatting for LLM reasoning.
• The framework demonstrates resilience to tool failures, with performance improvements through feedback-driven weight updates that suppress low-quality tools.
• Token consumption decreases as transition weights are scaled up, suggesting a potential optimization pathway for tool learning systems.
• The method maintains strong performance on complex, multi-task datasets like ToolBench, outperforming existing prompting-based and fine-tuned approaches in scalability and adaptability.