[Paper Review] xLAM: A Family of Large Action Models to Empower AI Agent Systems

Jianguo Zhang, Tian Lan+20 more|arXiv (Cornell University)|Sep 5, 2024

Multi-Agent Systems and Negotiation4 citations

TL;DR

xLAM is a family of large action models (1B to 8x22B parameters) trained via a scalable, flexible pipeline that unifies, augments, and synthesizes diverse agent datasets to enhance generalization and performance in autonomous AI agent tasks. The models achieve state-of-the-art results, securing 1st place on the Berkeley Function-Calling Leaderboard, outperforming GPT-4, Claude-3, and other leading models in tool use and function calling.

ABSTRACT

Autonomous agents powered by large language models (LLMs) have attracted significant research interest. However, the open-source community faces many challenges in developing specialized models for agent tasks, driven by the scarcity of high-quality agent datasets and the absence of standard protocols in this area. We introduce and publicly release xLAM, a series of large action models designed for AI agent tasks. The xLAM series includes five models with both dense and mixture-of-expert architectures, ranging from 1B to 8x22B parameters, trained using a scalable, flexible pipeline that unifies, augments, and synthesizes diverse datasets to enhance AI agents' generalizability and performance across varied environments. Our experimental results demonstrate that xLAM consistently delivers exceptional performance across multiple agent ability benchmarks, notably securing the 1st position on the Berkeley Function-Calling Leaderboard, outperforming GPT-4, Claude-3, and many other models in terms of tool use. By releasing the xLAM series, we aim to advance the performance of open-source LLMs for autonomous AI agents, potentially accelerating progress and democratizing access to high-performance models for agent tasks. Models are available at https://huggingface.co/collections/Salesforce/xlam-models-65f00e2a0a63bbcd1c2dade4

Motivation & Objective

• To address the scarcity of high-quality, diverse, and standardized datasets for training open-source LLMs in agent tasks.
• To overcome the limitations of existing open-source agent datasets, which suffer from format heterogeneity, low quality, and lack of diversity.
• To develop a scalable, flexible data pipeline that unifies, augments, and synthesizes diverse agent datasets to improve model generalization.
• To train and release a series of large action models (xLAM) that outperform existing open-source and proprietary models in agent benchmarks.
• To democratize access to high-performance agent models by releasing the xLAM series and associated training pipeline.

Proposed method

• A multi-stage data processing pipeline that unifies diverse agent datasets across multiple environments into a single, standardized format.
• Data augmentation via instruction tuning, prompt engineering, and synthetic data generation to increase diversity and reduce overfitting.
• A multi-stage data quality verification system using automated checks, error categorization, and iterative feedback loops to clean hallucinated or incorrect actions.
• Data synthesis using large language models to generate high-quality, diverse, and realistic agent trajectories, especially for rare or underrepresented tasks.
• Model training using supervised fine-tuning (SFT) and direct preference optimization (DPO) with both full fine-tuning and LoRA adapters for efficiency.
• Iterative evaluation and feedback loop: model performance results are used to diagnose data quality issues and guide further data cleaning and augmentation.

Experimental results

Research questions

• RQ1 How can diverse, heterogeneous, and low-quality open-source agent datasets be unified and enhanced to improve model generalization across environments?
• RQ2 To what extent can synthetic data generation improve the performance of small and large language models in agent tasks?
• RQ3 Can a scalable and flexible data pipeline significantly improve the performance of open-source LLMs on agent benchmarks compared to existing approaches?
• RQ4 What is the impact of data cleaning and augmentation on model performance in complex, multi-turn agent interactions?
• RQ5 Can smaller models (1B–7B parameters) achieve competitive performance with much larger models (e.g., 8x22B) through high-quality data curation?

Key findings

• xLAM-8x22B achieved the top rank on the Berkeley Function-Calling Leaderboard v2 (cutoff 09/03/2024), securing 1st place with an overall accuracy of 86.00%.
• xLAM-7b-r achieved 80.33% overall accuracy, ranking 14th and outperforming larger models like GPT-4 and GPT-4o.
• xLAM-1b-fc-r achieved 75.43% accuracy, ranking 32nd and surpassing larger models such as Claude-3-Opus (FC) and GPT-3.5-Turbo.
• Data augmentation improved performance by 2.3% on ToolBench, 5.8% on Webshop, and 18.3% on ToolQuery compared to raw data.
• Data cleaning further boosted performance by 23.4% on ToolQuery, demonstrating the critical role of quality verification in the pipeline.
• The models generalized well to unseen, real-world data from the live BFCL-v2 benchmark, indicating strong robustness and transferability.