[Paper Review] Agents in Software Engineering: Survey, Landscape, and Vision

Yanlin Wang, Wanjun Zhong+8 more|arXiv (Cornell University)|Sep 13, 2024

Multi-Agent Systems and Negotiation4 citations

TL;DR

This paper presents the first comprehensive survey of LLM-based agents in software engineering, introducing a three-module framework—perception, memory, and action—to systematize agent design. It identifies key challenges such as multimodal perception gaps, hallucinations, inefficient multi-agent collaboration, and lack of domain-specific knowledge bases, while proposing future research opportunities to bridge LLM agents and SE through mutual advancement.

ABSTRACT

In recent years, Large Language Models (LLMs) have achieved remarkable success and have been widely used in various downstream tasks, especially in the tasks of the software engineering (SE) field. We find that many studies combining LLMs with SE have employed the concept of agents either explicitly or implicitly. However, there is a lack of an in-depth survey to sort out the development context of existing works, analyze how existing works combine the LLM-based agent technologies to optimize various tasks, and clarify the framework of LLM-based agents in SE. In this paper, we conduct the first survey of the studies on combining LLM-based agents with SE and present a framework of LLM-based agents in SE which includes three key modules: perception, memory, and action. We also summarize the current challenges in combining the two fields and propose future opportunities in response to existing challenges. We maintain a GitHub repository of the related papers at: https://github.com/DeepSoftwareAnalytics/Awesome-Agent4SE.

Motivation & Objective

• To systematically survey and categorize existing research on LLM-based agents in software engineering.
• To establish a unified conceptual framework for LLM-based agents in SE, comprising perception, memory, and action modules.
• To identify critical challenges in integrating LLM-based agents with SE tasks, including hallucinations, lack of code-specific knowledge bases, and inefficient multi-agent collaboration.
• To propose future research directions that leverage SE technologies to enhance agent capabilities and vice versa.
• To curate and maintain a public GitHub repository of relevant papers for the research community.

Proposed method

• Collected and filtered 115 papers on LLM-based agents in software engineering through systematic data collection and quality assessment.
• Proposed a three-module framework for LLM-based agents in SE: perception (handling multimodal inputs), memory (semantic, episodic, procedural), and action (internal and external actions).
• Analyzed existing works through the lens of the proposed framework to identify gaps and trends in agent design and application.
• Identified challenges by examining limitations in perception beyond text, reasoning over complex SE tasks, and lack of external knowledge bases.
• Explored bidirectional synergies between SE and agent research, including using SE techniques (e.g., testing, version control) to improve agent reliability and performance.
• Proposed future research directions based on identified challenges, such as developing code-specific knowledge bases and optimizing multi-agent communication.

Experimental results

Research questions

• RQ1 How are LLM-based agents currently being applied in software engineering tasks, and what are the dominant patterns in their design?
• RQ2 What are the core components of LLM-based agents in SE, and how do they align with the perception-memory-action framework?
• RQ3 What are the major challenges hindering the effective integration of LLM-based agents into SE workflows?
• RQ4 How can SE-specific technologies improve the robustness, efficiency, and reliability of LLM-based agents?
• RQ5 What opportunities exist for mutual advancement between the LLM agent and software engineering fields?

Key findings

• The perception module in LLM-based agents for SE is predominantly limited to textual input, with minimal exploration of other modalities such as visual or auditory inputs.
• The memory module in current agents includes semantic, episodic, and procedural memory, but lacks standardized implementation and integration across systems.
• The action module involves both internal reasoning and external interactions, but external actions often rely on basic function calls and HTTP interactions, lacking advanced tool integration.
• A significant gap exists in the availability of authoritative, code-specific knowledge bases to serve as external retrieval sources for agents in SE.
• Hallucinations in LLM-based agents—such as inventing non-existent APIs—are a major issue that undermines reliability and can be mitigated through both LLM and agent-level optimizations.
• Multi-agent collaboration in SE tasks faces high computational and communication overhead, indicating a need for more efficient coordination and resource management techniques.