[논문 리뷰] Towards an Understanding of Large Language Models in Software Engineering Tasks

Zibin Zheng, Kaiwen Ning외 6명|arXiv (Cornell University)|2023. 08. 22.

Topic Modeling인용 수 10

한 줄 요약

이 논문은 대형 언어 모델(LLMs)이 소프트웨어 엔지니어링에 적용되는 방식에 대한 최초의 체계적 고찰로, seven task types를 분류하고 LLM이 어디에서 잘 수행하는지 평가한다. 또한 123 studies from six databases를 분석하여 동향과 효과를 파악한다.

ABSTRACT

Large Language Models (LLMs) have drawn widespread attention and research due to their astounding performance in text generation and reasoning tasks. Derivative products, like ChatGPT, have been extensively deployed and highly sought after. Meanwhile, the evaluation and optimization of LLMs in software engineering tasks, such as code generation, have become a research focus. However, there is still a lack of systematic research on applying and evaluating LLMs in software engineering. Therefore, this paper comprehensively investigate and collate the research and products combining LLMs with software engineering, aiming to answer two questions: (1) What are the current integrations of LLMs with software engineering? (2) Can LLMs effectively handle software engineering tasks? To find the answers, we have collected related literature as extensively as possible from seven mainstream databases and selected 123 timely papers published starting from 2022 for analysis. We have categorized these papers in detail and reviewed the current research status of LLMs from the perspective of seven major software engineering tasks, hoping this will help researchers better grasp the research trends and address the issues when applying LLMs. Meanwhile, we have also organized and presented papers with evaluation content to reveal the performance and effectiveness of LLMs in various software engineering tasks, guiding researchers and developers to optimize.

연구 동기 및 목표

• Survey the current landscape of integrating LLMs with software engineering tasks.
• Categorize existing work into seven software engineering task types.
• Assess whether LLMs improve performance on software engineering tasks and why.
• Provide guidance for researchers to address challenges in applying LLMs to software engineering.

제안 방법

• Literature search across six databases: ACM DL, IEEE Xplore, dblp, Elsevier Science Direct, Google Scholar, arXiv.
• Card sorting (closed) to identify relevant vs. irrelevant papers.
• Exclusion of non-English, theses, keynote papers, non-LLM, non-software-engineering works, duplicates, and pre-2022 studies.
• Data analysis involving reading papers to answer two research questions about applications and performance.

실험 결과

연구 질문

• RQ1 RQ1: What are the current works focusing on combining LLMs with software engineering?
• RQ2 RQ2: Can LLMs truly help better perform current software engineering tasks?

주요 결과

• LLMs show strength in syntax-related tasks such as code summarization and repair.
• LLMs are less satisfactory for semantics-heavy tasks like code generation and vulnerability detection, though progress continues with model iterations.
• A total of 123 relevant papers were identified and categorized into seven software engineering tasks.
• Code generation is the most studied category (24 papers), while code translation is the least (3 papers).
• The paper provides a structured view of the current status, applications, and evaluation content to guide optimization.
• Emergent abilities from scaling (e.g., in-context learning, instruction following) are discussed as contributing factors to LLM performance in SE tasks.