[Paper Review] MarsCode Agent: AI-native Automated Bug Fixing

Yizhou Liu, Pengfei Gao+5 more|arXiv (Cornell University)|Sep 2, 2024

Scientific Computing and Data ManagementDecision Sciences3 citations

TL;DR

MarsCode Agent is an LLM-native framework for automated, end-to-end software bug fixing that combines multi-agent planning, code knowledge graphs, language server protocols, and containerized debugging to localize faults and generate validated patches. It achieved a 34% solve rate on the SWE-bench Lite benchmark, outperforming existing tools in file and code snippet localization accuracy.

ABSTRACT

Recent advances in large language models (LLMs) have shown significant potential to automate various software development tasks, including code completion, test generation, and bug fixing. However, the application of LLMs for automated bug fixing remains challenging due to the complexity and diversity of real-world software systems. In this paper, we introduce MarsCode Agent, a novel framework that leverages LLMs to automatically identify and repair bugs in software code. MarsCode Agent combines the power of LLMs with advanced code analysis techniques to accurately localize faults and generate patches. Our approach follows a systematic process of planning, bug reproduction, fault localization, candidate patch generation, and validation to ensure high-quality bug fixes. We evaluated MarsCode Agent on SWE-bench, a comprehensive benchmark of real-world software projects, and our results show that MarsCode Agent achieves a high success rate in bug fixing compared to most of the existing automated approaches.

Motivation & Objective

• To address the challenge of automating real-world software bug fixing in complex, large-scale codebases using LLMs.
• To reduce reliance on heuristic-based or rule-driven approaches by enabling autonomous, context-aware fault localization and patch generation.
• To improve the reliability and scalability of automated program repair through integrated static and dynamic analysis workflows.
• To enhance code understanding and modification accuracy using code knowledge graphs and language server protocol integrations.
• To enable end-to-end, human-free execution of the software repair pipeline, including patch validation in isolated environments.

Proposed method

• MarsCode Agent employs a multi-agent collaboration framework that dynamically assigns static or dynamic repair pipelines based on problem characteristics.
• It leverages code knowledge graphs and Language Server Protocol (LSP) integrations to enable precise code entity retrieval, definition/reference navigation, and contextual code understanding.
• The system uses conflict-based code edit descriptions and static syntax checking to generate well-formed, syntactically correct code patches.
• For dynamic debugging, it utilizes a Docker-based sandbox environment to reproduce defects, inject logs, and execute test frameworks—mimicking human developer debugging workflows.
• The agent pipeline includes: planning, bug reproduction, fault localization, candidate patch generation, and validation, ensuring correctness and minimizing regressions.
• It combines LLM reasoning with structured tool use to guide systematic exploration of codebases and repair strategies.

Experimental results

Research questions

• RQ1 Can an LLM-based agent achieve high-accuracy, end-to-end automated bug fixing in real-world software projects?
• RQ2 How effective is a multi-agent framework in dynamically selecting between static and dynamic repair strategies for diverse bug types?
• RQ3 To what extent can code knowledge graphs and LSP integration improve fault localization precision in complex codebases?
• RQ4 What is the impact of sandboxed dynamic debugging on patch success rates compared to purely static repair?
• RQ5 How does MarsCode Agent compare to existing agents in terms of file and code snippet localization accuracy on real-world benchmarks?

Key findings

• MarsCode Agent achieved a 34% solve rate on the SWE-bench Lite benchmark, successfully resolving 102 out of 300 real-world software engineering tasks.
• It demonstrated a file localization precision of 88.3%, correctly identifying the target file in 265 out of 300 instances, outperforming prior state-of-the-art tools.
• The system achieved a code snippet localization precision of 68.7%, successfully identifying the modification target in 206 instances.
• Of the 300 instances, 28% were routed to dynamic debugging, where 38.1% of those (32/84) were successfully resolved, showing higher success than static repair.
• Static repair succeeded in 70 out of 216 cases (32.4% success rate), indicating that dynamic debugging is more effective for complex or context-sensitive bugs.
• The agent’s code retrieval and localization capabilities were superior to existing traceable solutions, including CodeR, Moatless, and Agentless, as shown in comparative evaluations.