[论文解读] AutoDev: Automated AI-Driven Development

Michele Tufano, Anisha Agarwal+3 more|arXiv (Cornell University)|Mar 13, 2024

Scientific Computing and Data Management被引用 12

一句话总结

AutoDev 是一个完全自动化的 AI 驱动的软件开发框架，能够在 Docker 封装的环境中自主规划并执行任务，支持代码编辑、构建、测试和 git 操作，以实现用户定义的目标。它在 HumanEval 上展示出强大的代码和测试生成性能，无需额外训练。

ABSTRACT

The landscape of software development has witnessed a paradigm shift with the advent of AI-powered assistants, exemplified by GitHub Copilot. However, existing solutions are not leveraging all the potential capabilities available in an IDE such as building, testing, executing code, git operations, etc. Therefore, they are constrained by their limited capabilities, primarily focusing on suggesting code snippets and file manipulation within a chat-based interface. To fill this gap, we present AutoDev, a fully automated AI-driven software development framework, designed for autonomous planning and execution of intricate software engineering tasks. AutoDev enables users to define complex software engineering objectives, which are assigned to AutoDev's autonomous AI Agents to achieve. These AI agents can perform diverse operations on a codebase, including file editing, retrieval, build processes, execution, testing, and git operations. They also have access to files, compiler output, build and testing logs, static analysis tools, and more. This enables the AI Agents to execute tasks in a fully automated manner with a comprehensive understanding of the contextual information required. Furthermore, AutoDev establishes a secure development environment by confining all operations within Docker containers. This framework incorporates guardrails to ensure user privacy and file security, allowing users to define specific permitted or restricted commands and operations within AutoDev. In our evaluation, we tested AutoDev on the HumanEval dataset, obtaining promising results with 91.5% and 87.8% of Pass@1 for code generation and test generation respectively, demonstrating its effectiveness in automating software engineering tasks while maintaining a secure and user-controlled development environment.

研究动机与目标

• 推动基于 AI 的自主软件开发，超越代码片段建议。
• 使 AI 代理能够在拥有完整仓库访问权限的情况下执行复杂的软件工程任务。
• 提供带有保护措施和权限的安全、可配置环境。
• 在代码生成和测试生成基准测试中展示有效性。

提出的方法

• 四组件架构：会话管理器、工具库、代理调度器和评估环境。
• 通过 YAML 配置规则和动作，以自定义代理的权限与能力。
• 代理（LLMs/SLMs）提出仓库操作，由 Agent Scheduler 协调。
• 在基于 Docker 的评估环境中对操作进行安全执行，输出反馈回会话。
• 命令工具包括文件编辑、检索、构建/执行、测试和 git 操作；在执行前对动作进行解析与验证。

Figure 1. AutoDev enables an AI Agent to achieve a given objective by performing several actions within the repository. The Eval Environment executes the suggested operations, providing the AI Agent with the resulting outcome. In the conversation, purple messages are from the AI agent, while blue me

实验结果

研究问题

• RQ1 RQ1：在 HumanEval 数据集上的代码生成有效性（Pass@1）如何？
• RQ2 RQ2：在 HumanEval 的测试生成（Pass@1 与覆盖率）上 AutoDev 的有效性如何？
• RQ3 RQ3：在完成任务的效率方面（步骤数量、令牌数量和命令分布）AutoDev 的表现如何？

主要发现

方法	模型	额外训练	Pass@1
语言代理树搜索	GPT-4	✓	94.4
AutoDev	GPT-4	×	91.5
反思	GPT-4	✓	91.0
零样本（基线）	GPT-4	×	67.0
通过	覆盖率	总体	覆盖率
人类	-	100	99.4	99.4
AutoDev	GPT-4	87.8	99.3	88.8
零样本（基线）	GPT-4	75	99.3	74

• AutoDev 在 HumanEval 的代码生成上实现 Pass@1 91.5%，在排行榜上名列第二，无需额外训练数据。
• AutoDev 在修改后的 HumanEval 的测试生成上实现 Pass@1 87.8%，覆盖率与人工编写的测试相当（99.3% 对 99.4%）。
• AutoDev 将 GPT-4 在代码生成任务的表现从 67% 提升到 91.5%，相对提升约 30%。
• 在代码生成方面，AutoDev 每个任务平均使用 5.5 个命令（其中 1.8 写入，1.7 测试，0.92 停止）；在测试生成方面，平均约使用 6.5 个命令。
• 该方法通过在安全的基于 Docker 的评估环境中进行迭代、自治的循环来执行任务，并具有权限和可重复性的护栏。
• AutoDev 展示了多代理协作潜力以及面向未来改进的人机交互能力（对话/提问）。

Figure 2. Overview of the AutoDev Framework: The user initiates the process by defining the objective to be achieved. The Conversation Manager initializes the conversation and settings. The Agent Scheduler orchestrates AI agents to collaborate on the task and forwards their commands to the Conversat