[Paper Review] TaskWeaver: A Code-First Agent Framework

Bo Qiao, Liqun Li+17 more|arXiv (Cornell University)|Nov 29, 2023

Topic Modeling4 citations

TL;DR

TaskWeaver is a code-first framework that transforms user requests into executable code, treating plugins as callable functions to enable flexible, secure, and domain-aware LLM-powered agents. It supports rich data structures like DataFrames, enables dynamic plugin selection, and leverages LLMs for complex logic, significantly improving task handling in data analytics scenarios with high reliability and security.

ABSTRACT

Large Language Models (LLMs) have shown impressive abilities in natural language understanding and generation, leading to their widespread use in applications such as chatbots and virtual assistants. However, existing LLM frameworks face limitations in handling domain-specific data analytics tasks with rich data structures. Moreover, they struggle with flexibility to meet diverse user requirements. To address these issues, TaskWeaver is proposed as a code-first framework for building LLM-powered autonomous agents. It converts user requests into executable code and treats user-defined plugins as callable functions. TaskWeaver provides support for rich data structures, flexible plugin usage, and dynamic plugin selection, and leverages LLM coding capabilities for complex logic. It also incorporates domain-specific knowledge through examples and ensures the secure execution of generated code. TaskWeaver offers a powerful and flexible framework for creating intelligent conversational agents that can handle complex tasks and adapt to domain-specific scenarios. The code is open sourced at https://github.com/microsoft/TaskWeaver/.

Motivation & Objective

• To address the limitations of existing LLM frameworks in handling rich data structures like DataFrames and nested objects in data analytics tasks.
• To enable flexible, dynamic plugin usage and integration with custom code generation for ad-hoc user queries.
• To incorporate domain-specific knowledge systematically through examples, improving planning and code generation accuracy.
• To ensure secure execution of generated code by restricting high-risk operations such as file deletion and secret key access.
• To provide a developer-friendly, extensible framework that supports stateful, iterative interactions across multiple conversation rounds.

Proposed method

• Convert user natural language requests into executable Python code using LLMs, treating all actions—including plugin calls—as code functions.
• Support rich data structures (e.g., pandas DataFrames) as first-class objects that persist and are passed across plugin and code execution steps.
• Implement stateful execution by maintaining conversation context and data state across multiple chat rounds.
• Enable dynamic plugin selection and usage, allowing agents to choose appropriate plugins based on task requirements.
• Integrate domain-specific knowledge via few-shot examples to guide planning and code generation.
• Enforce security through a customizable restricted operation list that blocks high-risk actions like file deletion and environment variable access.

Experimental results

Research questions

• RQ1 How can LLM-powered agents effectively handle complex data analytics tasks involving rich data structures such as DataFrames?
• RQ2 To what extent can a code-first approach improve flexibility and accuracy in plugin and custom code integration for diverse user requests?
• RQ3 How can domain-specific knowledge be systematically embedded into the agent’s planning and code generation process?
• RQ4 What mechanisms ensure secure execution of dynamically generated code in agent frameworks?
• RQ5 How does stateful, multi-round interaction improve the reliability and correctness of agent behavior in iterative data analysis workflows?

Key findings

• TaskWeaver successfully handles complex data analytics tasks involving multiple plugins and rich data structures, such as loading, analyzing, and visualizing time series data from SQL databases.
• The framework enables seamless state persistence across conversation rounds, allowing the agent to reference data from previous steps without re-prompting or reprocessing.
• Through auto-correction mechanisms, TaskWeaver identifies and fixes code errors—such as incorrect column selection in mean calculations—by revising code based on execution feedback.
• The system prevents high-risk operations like file deletion and secret key leakage by enforcing a customizable restricted operation list, enhancing security.
• TaskWeaver demonstrates robust performance in real-world case studies, including anomaly detection and data analysis, by combining LLM reasoning with executable code and plugin orchestration.
• The framework supports flexible plugin usage and dynamic selection, enabling agents to adapt to both standard and ad-hoc queries without requiring pre-built plugins for every scenario.