[Paper Review] Large Language Models for Software Engineering: A Systematic Literature Review

Xinyi Hou, Yanjie Zhao+8 more|arXiv (Cornell University)|Aug 21, 2023

Software Engineering ResearchComputer Science111 citations

TL;DR

A systematic literature review analyzing 229 papers (2017–2023) on applying large language models to software engineering, categorizing models, data practices, optimization/evaluation strategies, and SE tasks.

ABSTRACT

Large Language Models (LLMs) have significantly impacted numerous domains, including Software Engineering (SE). Many recent publications have explored LLMs applied to various SE tasks. Nevertheless, a comprehensive understanding of the application, effects, and possible limitations of LLMs on SE is still in its early stages. To bridge this gap, we conducted a systematic literature review (SLR) on LLM4SE, with a particular focus on understanding how LLMs can be exploited to optimize processes and outcomes. We select and analyze 395 research papers from January 2017 to January 2024 to answer four key research questions (RQs). In RQ1, we categorize different LLMs that have been employed in SE tasks, characterizing their distinctive features and uses. In RQ2, we analyze the methods used in data collection, preprocessing, and application, highlighting the role of well-curated datasets for successful LLM for SE implementation. RQ3 investigates the strategies employed to optimize and evaluate the performance of LLMs in SE. Finally, RQ4 examines the specific SE tasks where LLMs have shown success to date, illustrating their practical contributions to the field. From the answers to these RQs, we discuss the current state-of-the-art and trends, identifying gaps in existing research, and flagging promising areas for future study. Our artifacts are publicly available at https://github.com/xinyi-hou/LLM4SE_SLR.

Motivation & Objective

• Map which LLMs have been used for SE tasks and how they are categorized by architecture and features.
• Analyze data collection, preprocessing, and representation practices for LLM4SE studies.
• Identify optimization and evaluation strategies used for LLMs in SE.
• Identify SE tasks where LLMs have shown effectiveness and derive trends, gaps, and future directions.

Proposed method

• Followed a Kitchenham-style systematic literature review process (planning, conducting, analyzing).
• Built a Quasi-Gold Standard from manually identified relevant papers, then conducted automated searches and snowballing to reach comprehensiveness.
• Applied explicit inclusion/exclusion criteria and a 10-item quality assessment checklist to select high-quality primary studies.
• Conducted data extraction on SE task categories, LLM categories, data handling, optimization algorithms, evaluation metrics, and SE activities.
• Performed descriptive analyses and trend analyses across publication venues, years, and architectures (encoder-only, encoder-decoder, decoder-only).
• Synthesized results to map the state-of-the-art, challenges, and future research directions.

Experimental results

Research questions

• RQ1 RQ1: What LLMs have been employed to date to solve SE tasks?
• RQ2 RQ2: How are SE-related datasets collected, preprocessed, and used in LLMs?
• RQ3 RQ3: What techniques are used to optimize and evaluate LLM4SE?
• RQ4 RQ4: What SE tasks have been effectively addressed to date using LLM4SE?

Key findings

• The study is the first comprehensive SLR on LLM-based solutions for SE, analyzing 229 papers from 2017–2023.
• More than 50 different LLMs were used for SE tasks in the collected literature.
• Encoder-only, encoder-decoder, and decoder-only LLMs are used, with decoder-only dominance emerging in 2023.
• The literature shows diverse data handling practices and a variety of optimization and evaluation approaches tailored to SE tasks.
• SE tasks span 55 distinct activities, grouped into six core SE activities (requirements, design, development, QA, maintenance, management).
• A rapid growth trend is evident, with publications rising sharply in 2022–early 2023, and a substantial share of works appearing on arXiv (reflecting ongoing rapid development).
• The review identifies challenges and proposes directions for future research in LLM4SE, including model choice, data handling, fine-tuning, evaluation, and deployment considerations.