[Paper Review] HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face

Yongliang Shen, Kaitao Song+4 more|arXiv (Cornell University)|Mar 30, 2023

Topic Modeling264 citations

TL;DR

HuggingGPT uses ChatGPT as a controller to orchestrate Hugging Face models, enabling multi-modal AI task solving via a language-based planning, selection, execution, and response-generation pipeline.

ABSTRACT

Solving complicated AI tasks with different domains and modalities is a key step toward artificial general intelligence. While there are numerous AI models available for various domains and modalities, they cannot handle complicated AI tasks autonomously. Considering large language models (LLMs) have exhibited exceptional abilities in language understanding, generation, interaction, and reasoning, we advocate that LLMs could act as a controller to manage existing AI models to solve complicated AI tasks, with language serving as a generic interface to empower this. Based on this philosophy, we present HuggingGPT, an LLM-powered agent that leverages LLMs (e.g., ChatGPT) to connect various AI models in machine learning communities (e.g., Hugging Face) to solve AI tasks. Specifically, we use ChatGPT to conduct task planning when receiving a user request, select models according to their function descriptions available in Hugging Face, execute each subtask with the selected AI model, and summarize the response according to the execution results. By leveraging the strong language capability of ChatGPT and abundant AI models in Hugging Face, HuggingGPT can tackle a wide range of sophisticated AI tasks spanning different modalities and domains and achieve impressive results in language, vision, speech, and other challenging tasks, which paves a new way towards the realization of artificial general intelligence.

Motivation & Objective

• Demonstrate a framework where an LLM coordinates external expert models to solve complex AI tasks across multiple modalities.
• Show how language can serve as a universal interface to connect LLMs with heterogeneous AI models.
• Propose a four-stage pipeline (planning, selection, execution, response) and validate its effectiveness on language, vision, and speech tasks.
• Analyze how task planning quality and model selection impact overall task solving and user satisfaction.

Proposed method

• Four-stage pipeline: task planning, model selection, task execution, and response generation.
• Task planning uses LLM prompts with a standardized JSON task format and demonstrations to decompose user requests and establish task dependencies.
• Model selection relies on in-context assignment using descriptions from Hugging Face and ranks candidates by downloads to choose top-K models.
• Task execution feeds task arguments (and dynamic resources) to selected models, handling resource dependencies with a <resource-task_id> mechanism and enabling parallel execution where possible.
• Response generation aggregates model outputs into a coherent user-facing answer, including a summary of the workflow and confidence in the results.

Experimental results

Research questions

• RQ1 Can an LLM effectively plan and decompose user requests into a structured set of subtasks with dependencies?
• RQ2 How well does an LLM select appropriate external models using descriptions from a public model hub (Hugging Face)?
• RQ3 What is the performance and behavior of the end-to-end HuggingGPT pipeline across language, vision, speech, and cross-modality tasks?
• RQ4 How do demonstrations and prompt design influence the quality of task planning and model selection?

Key findings

• HuggingGPT can automatically plan tasks, assign expert models, execute subtasks, and generate a final user-facing response across multiple modalities.
• Model selection based on Hugging Face descriptions enables flexible integration of diverse models without changing core prompts.
• Task planning quality correlates with LLM capability; GPT-3.5 shows strong planning among evaluated open-source LLMs, with GPT-4 generally performing better on graph tasks and human annotations revealing remaining gaps.
• Human evaluation indicates GPT-3.5 outperforms Alpaca-13b and Vicuna-13b in planning and model selection stages, while GPT-4 demonstrates stronger performance but still leaves room relative to human judgments.
• Ablation studies show demonstrations and their variety modestly influence planning performance, with diminishing returns beyond a small number of examples.