[论文解读] Practices and Challenges of Using GitHub Copilot: An Empirical Study

Beiqi Zhang, Peng Liang+3 more|arXiv (Cornell University)|Mar 15, 2023

Software Engineering Research被引用 5

一句话总结

本实证研究通过分析169篇Stack Overflow帖子和655个GitHub Discussions，调查了GitHub Copilot在现实世界中的使用实践与挑战。研究发现，开发者主要在Visual Studio Code中使用JavaScript和Python进行数据处理任务，尽管在集成和代码质量方面存在挑战，但依然能从有用的代码生成中受益，凸显Copilot作为一把双刃剑，其采用需谨慎决策。

ABSTRACT

With the advances in machine learning, there is a growing interest in AI-enabled tools for autocompleting source code. GitHub Copilot, also referred to as the "AI Pair Programmer", has been trained on billions of lines of open source GitHub code, and is one of such tools that has been increasingly used since its launch on June 2021. However, little effort has been devoted to understanding the practices and challenges of using Copilot in programming with auto-completed source code. To this end, we conducted an empirical study by collecting and analyzing the data from Stack Overflow (SO) and GitHub Discussions. More specifically, we searched and manually collected 169 SO posts and 655 GitHub discussions related to the usage of Copilot. We identified the programming languages, IDEs, technologies used with Copilot, functions implemented, benefits, limitations, and challenges when using Copilot. The results show that when practitioners use Copilot: (1) The major programming languages used with Copilot are JavaScript and Python, (2) the main IDE used with Copilot is Visual Studio Code, (3) the most common used technology with Copilot is Node.js, (4) the leading function implemented by Copilot is data processing, (5) the significant benefit of using Copilot is useful code generation, and (6) the main limitation encountered by practitioners when using Copilot is difficulty of integration. Our results suggest that using Copilot is like a double-edged sword, which requires developers to carefully consider various aspects when deciding whether or not to use it. Our study provides empirically grounded foundations and basis for future research on the role of Copilot as an AI pair programmer in software development.

研究动机与目标

• 了解软件开发者在真实编程环境中实际如何使用GitHub Copilot。
• 识别与Copilot最常搭配使用的编程语言、IDE和技术。
• 揭示开发者在使用Copilot时面临的主要优势、局限性和挑战。
• 为AI辅助编程的未来研究和工具改进提供实证基础的见解。

提出的方法

• 使用'copilot'作为搜索词，收集并分析了169篇Stack Overflow帖子和655个GitHub Discussions。
• 由多位作者进行手动数据标注、提取与分析，以确保一致性和减少偏差。
• 通过初步标注阶段使用Cohen’s Kappa（0.773）验证标注者间的一致性。
• 按编程语言、IDE、技术、实现功能、优势、局限性和挑战对响应内容进行分类。
• 通过作者间的迭代审查与讨论，解决分歧并确保结果可靠性。
• 公开发布完整数据集，以支持复现与验证。

Figure 1 : Overview of the research process

实验结果

研究问题

• RQ1 与GitHub Copilot最常搭配使用的编程语言、IDE和技术是什么？
• RQ2 使用Copilot代码建议最常实现的函数类型有哪些？
• RQ3 开发者报告使用Copilot时的主要优势是什么？
• RQ4 开发者在使用Copilot时面临的主要局限性和挑战是什么？
• RQ5 Copilot的优势与局限性在实践中如何相互作用？

主要发现

• JavaScript和Python是与GitHub Copilot最常使用的编程语言。
• Visual Studio Code是与Copilot搭配使用的主导IDE。
• Node.js是在开发工作流中与Copilot最常搭配使用的技术。
• 数据处理是使用Copilot代码建议最常实现的功能。
• 开发者报告的最重要优势是生成有用代码，特别是用于样板代码和重复性任务。
• 主要局限性在于难以将Copilot集成到现有开发工作流中，常导致挫败感和生产力下降。

Figure 2 : Programming languages, IDEs, technologies, and implemented functions of using Copilot (results of RQ1 to RQ4)