[論文レビュー] AutoDev: Automated AI-Driven Development

Michele Tufano, Anisha Agarwal+3 more|arXiv (Cornell University)|Mar 13, 2024

Scientific Computing and Data Management被引用数 12

ひとこと要約

AutoDev は完全自動の AI 主導のソフトウェア開発フレームワークで、Docker-contained 環境内でタスクを自律的に計画・実行し、コード編集、ビルド、テスト、そして git 操作を可能にし、ユーザー定義の目標を達成します。追加のトレーニングなしで HumanEval で強力なコードおよびテスト生成性能を示します。

ABSTRACT

The landscape of software development has witnessed a paradigm shift with the advent of AI-powered assistants, exemplified by GitHub Copilot. However, existing solutions are not leveraging all the potential capabilities available in an IDE such as building, testing, executing code, git operations, etc. Therefore, they are constrained by their limited capabilities, primarily focusing on suggesting code snippets and file manipulation within a chat-based interface. To fill this gap, we present AutoDev, a fully automated AI-driven software development framework, designed for autonomous planning and execution of intricate software engineering tasks. AutoDev enables users to define complex software engineering objectives, which are assigned to AutoDev's autonomous AI Agents to achieve. These AI agents can perform diverse operations on a codebase, including file editing, retrieval, build processes, execution, testing, and git operations. They also have access to files, compiler output, build and testing logs, static analysis tools, and more. This enables the AI Agents to execute tasks in a fully automated manner with a comprehensive understanding of the contextual information required. Furthermore, AutoDev establishes a secure development environment by confining all operations within Docker containers. This framework incorporates guardrails to ensure user privacy and file security, allowing users to define specific permitted or restricted commands and operations within AutoDev. In our evaluation, we tested AutoDev on the HumanEval dataset, obtaining promising results with 91.5% and 87.8% of Pass@1 for code generation and test generation respectively, demonstrating its effectiveness in automating software engineering tasks while maintaining a secure and user-controlled development environment.

研究の動機と目的

• Motivate autonomous AI-driven software development beyond code snippet suggestions.
• Enable complex SE tasks to be executed by AI agents with full repository access.
• Provide a secure, configurable environment with guardrails and permissions.
• Demonstrate effectiveness on code generation and test generation benchmarks.

提案手法

• Four-component architecture: Conversation Manager, Tools Library, Agent Scheduler, and Evaluation Environment.
• Rule and action configuration via YAML to customize agent permissions and capabilities.
• Agents (LLMs/SLMs) propose repository actions and are orchestrated by the Agent Scheduler.
• Secure execution of actions inside Docker-based Evaluation Environment with outputs fed back into conversations.
• Command tools include file editing, retrieval, build/execution, testing, and git operations; actions are parsed and validated before execution.

Figure 1. AutoDev enables an AI Agent to achieve a given objective by performing several actions within the repository. The Eval Environment executes the suggested operations, providing the AI Agent with the resulting outcome. In the conversation, purple messages are from the AI agent, while blue me

実験結果

リサーチクエスチョン

• RQ1 RQ1: How effective is AutoDev in code generation on the HumanEval dataset (Pass@1)?
• RQ2 RQ2: How effective is AutoDev in test generation on HumanEval (Pass@1 and coverage)?
• RQ3 RQ3: How efficient is AutoDev in completing tasks (number of steps, tokens, and command distribution).

主な発見

] and an extra quote? Let's correct: The overall object should have fields: title, tldr, meta_description, objective, method, research_questions, key_findings, table_headers, table_rows. Not include

• AutoDev achieves Pass@1 of 91.5% for code generation on HumanEval, placing second on the leaderboard without extra training data.
• AutoDev achieves Pass@1 of 87.8% for test generation on modified HumanEval, with coverage comparable to human-written tests (99.3% vs 99.4%).
• AutoDev improves GPT-4 performance from 67% to 91.5% on code generation tasks, a 30% relative improvement.
• For code generation, AutoDev uses an average of 5.5 commands per task (including 1.8 write, 1.7 test, and 0.92 stop); for test generation, it uses about 6.5 commands on average.
• The approach executes tasks through iterative, autonomous cycles within a secure Docker-based Evaluation Environment, with guardrails for permissions and reproducibility.
• AutoDev demonstrates multi-agent collaboration potential and human-in-the-loop capabilities (talk/ask) for future enhancements.]
• table_headers: ["Approach", "Model", "Extra Training", "Pass@1"]
• table_rows: [["Language Agent Tree Search", "GPT-4", "✓", "94.4"], ["AutoDev", "GPT-4", "×", "91.5"], ["Reflexion", "GPT-4", "✓", "91.0"], ["zero-shot (baseline)", "GPT-4", "×", "67.0"], ["Passing", "Coverage", "Overall", "Coverage"], ["Human", "-", "100", "99.4", "99.4"], ["AutoDev", "GPT-4", "87.8", "99.3", "88.8"], ["zero-shot (baseline)", "GPT-4", "75", "99.3", "74"]]} }```json? The output is JSON-like. Replace final with proper JSON without extra backticks. But currently includes stray. We should provide valid JSON without trailing backticks. I'll adjust. Ensure table_headers and table_rows keys are strings; in this JSON, there is a mismatch: We included

Figure 2. Overview of the AutoDev Framework: The user initiates the process by defining the objective to be achieved. The Conversation Manager initializes the conversation and settings. The Agent Scheduler orchestrates AI agents to collaborate on the task and forwards their commands to the Conversat