[Paper Review] Generative Software Engineering

Yuan Huang, Yinan Chen+7 more|arXiv (Cornell University)|Mar 5, 2024

Advanced Software Engineering Methodologies4 citations

TL;DR

This paper presents a systematic literature review of generative tasks in software engineering powered by pre-trained models and large language models (LLMs), categorizing them across the software development lifecycle. It analyzes model architectures, datasets, evaluation metrics, and identifies key challenges and research gaps, offering a comprehensive roadmap for future work in LLM-based software engineering.

ABSTRACT

The rapid development of deep learning techniques, improved computational power, and the availability of vast training data have led to significant advancements in pre-trained models and large language models (LLMs). Pre-trained models based on architectures such as BERT and Transformer, as well as LLMs like ChatGPT, have demonstrated remarkable language capabilities and found applications in Software engineering. Software engineering tasks can be divided into many categories, among which generative tasks are the most concern by researchers, where pre-trained models and LLMs possess powerful language representation and contextual awareness capabilities, enabling them to leverage diverse training data and adapt to generative tasks through fine-tuning, transfer learning, and prompt engineering. These advantages make them effective tools in generative tasks and have demonstrated excellent performance. In this paper, we present a comprehensive literature review of generative tasks in SE using pre-trained models and LLMs. We accurately categorize SE generative tasks based on software engineering methodologies and summarize the advanced pre-trained models and LLMs involved, as well as the datasets and evaluation metrics used. Additionally, we identify key strengths, weaknesses, and gaps in existing approaches, and propose potential research directions. This review aims to provide researchers and practitioners with an in-depth analysis and guidance on the application of pre-trained models and LLMs in generative tasks within SE.

Motivation & Objective

• To address the lack of systematic literature reviews that bridge pre-trained models and LLMs in software engineering.
• To categorize generative software engineering tasks across the software development lifecycle.
• To analyze representative models, datasets, and evaluation metrics used in generative SE tasks.
• To identify limitations, challenges, and research gaps in current approaches.
• To propose actionable future research directions for LLMs in software engineering

Proposed method

• Conducted a systematic literature review focusing on generative tasks in software engineering using pre-trained models and LLMs.
• Categorized software engineering tasks into seven sub-tasks: requirements generation, code generation, code summarization, test generation, patch generation, code optimization, and code translation.
• Reviewed and compared key pre-trained models (e.g., BERT, GPT, XLNet) and LLMs (e.g., ChatGPT, GPT-4) across these tasks.
• Analyzed datasets and evaluation metrics used in existing studies to assess model performance.
• Synthesized findings on fine-tuning, prompt engineering, and transfer learning strategies in generative SE applications.
• Identified recurring challenges such as model instability, lack of code-specific fine-tuning, and insufficient evaluation of code-centric LLMs.

Experimental results

Research questions

• RQ1 How have pre-trained models evolved into large language models (LLMs) in the context of software engineering?
• RQ2 What are the distinct generative tasks in software engineering, and how are they distributed across the software development lifecycle?
• RQ3 Which pre-trained models and LLMs are most effective for different types of generative software engineering tasks?
• RQ4 What datasets and evaluation metrics are commonly used in current research, and how do they impact model assessment?
• RQ5 What are the key limitations and open challenges in applying LLMs to generative software engineering tasks?

Key findings

• LLMs such as GPT-4 outperform baseline models in code summarization and code translation from C# to Java when using task-specific prompts.
• Basic prompts often lead to suboptimal performance in code generation and cross-language translation, especially from Java to C#.
• Context learning and prompt engineering significantly improve LLM output quality, with user studies showing that iterative prompting enhances response relevance.
• Fine-tuning and prompt engineering are critical for adapting general-purpose LLMs to specific software engineering tasks.
• There is a lack of detailed evaluation for code-centric LLMs like Codex, with most studies relying on general models such as ChatGPT.
• Current research shows that LLMs struggle with consistency and reliability in certain tasks, indicating a need for better task-specific fine-tuning and evaluation frameworks.