[Paper Review] Harnessing LLMs for API Interactions: A Framework for Classification and Synthetic Data Generation

Chunhui Tao, Xiaojing Fan+1 more|arXiv (Cornell University)|Sep 18, 2024

Business Process Modeling and Analysis8 citations

TL;DR

The paper presents a framework that uses LLMs to classify natural language inputs into API calls and to generate synthetic datasets for evaluating API-classification performance across multiple LLMs. GPT-4 achieves the highest accuracy in API classification tasks among tested models.

ABSTRACT

As Large Language Models (LLMs) advance in natural language processing, there is growing interest in leveraging their capabilities to simplify software interactions. In this paper, we propose a novel system that integrates LLMs for both classifying natural language inputs into corresponding API calls and automating the creation of sample datasets tailored to specific API functions. By classifying natural language commands, our system allows users to invoke complex software functionalities through simple inputs, improving interaction efficiency and lowering the barrier to software utilization. Our dataset generation approach also enables the efficient and systematic evaluation of different LLMs in classifying API calls, offering a practical tool for developers or business owners to assess the suitability of LLMs for customized API management. We conduct experiments on several prominent LLMs using generated sample datasets for various API functions. The results show that GPT-4 achieves a high classification accuracy of 0.996, while LLaMA-3-8B performs much worse at 0.759. These findings highlight the potential of LLMs to transform API management and validate the effectiveness of our system in guiding model testing and selection across diverse applications.

Motivation & Objective

• Motivate enabling natural language interfaces for API interactions to lower technical barriers.
• Develop an end-to-end system that classifies NL prompts into API calls and executes them.
• Create a synthetic data generation pipeline to benchmark and compare LLMs on API classification tasks.
• Provide a scalable methodology for evaluating model suitability for customized API management.

Proposed method

• Two-component system: (1) API retrieval/classification pipeline mapping NL prompts to API calls with a defined API hierarchy; (2) dataset generation pipeline producing labeled synthetic NL prompts for API functions.
• Dataset generation uses batch prompting (100 synthetic queries per batch) to create diverse, labeled prompts in JSON, with manual label validation at 99.9% accuracy.
• Evaluation uses multiple LLMs (GPT-4, GPT-4o-mini, GPT-3.5-turbo, LLaMA3-70B, LLaMA3-8B, Gemini-1.5) to classify queries into API modules and functions.
• Metrics include Module-Level Classification Accuracy (MLC-Acc) and Function-Level Classification Accuracy (FLC-Acc).
• The dataset covers six API modules: Calculator, Notes, Weather, Email, Notification, Calendar, plus a Routes-Not-Exist module to test invalid calls.

Experimental results

Research questions

• RQ1 How accurately can state-of-the-art LLMs classify natural language prompts into predefined API modules and functions?
• RQ2 What is the impact of model size and architecture on API classification performance across diverse API modules?
• RQ3 Can synthetic data generation enable reliable benchmarking and model selection for API management tasks?
• RQ4 How does the proposed framework perform across simple versus complex API tasks?
• RQ5 What guidance can be provided for selecting LLMs for customized API management tasks?

Key findings

• GPT-4 achieves the highest overall module/function classification performance (MLC-Acc around 0.99 and FLC-Acc near 0.996 in reported results).
• LLaMA-3-70B also demonstrates strong performance (high MLC-Acc and FLC-Acc across modules).
• Smaller models (e.g., GPT-4o-mini, LLaMA3-8B) show notable drops in accuracy, especially on more complex modules, indicating model size significantly affects task performance.
• The dataset generation framework enables rapid, automated benchmarking of multiple LLMs for API classification tasks and helps identify the most suitable model per use case.
• The system demonstrates high-quality synthetic data generation with manual verification, enabling scalable evaluation of API-classification capabilities.