[论文解读] The Impact of Large Language Models on Open-source Innovation: Evidence from GitHub Copilot

Doron Yeverechyahu, Raveesh Mayya+1 more|arXiv (Cornell University)|Sep 12, 2024

Open Source Software Innovations被引用 7

一句话总结

本文使用围绕 GitHub Copilot 2021 年推出的自然实验，显示 Copilot 可用性使开源贡献增加 28–40%，其中增量贡献的增长高于实质性贡献，并分析情境与模型升级如何影响代码协作中的开发利用与探索。

ABSTRACT

Large Language Models (LLMs) have been shown to enhance individual productivity in guided settings. Whereas LLMs are likely to also transform innovation processes in a collaborative work setting, it is unclear what trajectory this transformation will follow. Innovation in these contexts encompasses both capability innovation that explores new possibilities by acquiring new competencies in a project and iterative innovation that exploits existing foundations by enhancing established competencies and improving project quality. Whether LLMs affect these two aspects of collaborative work and to what extent is an open empirical question. Open-source development provides an ideal setting to examine LLM impacts on these innovation types, as its voluntary and open/collaborative nature of contributions provides the greatest opportunity for technological augmentation. We focus on open-source projects on GitHub by leveraging a natural experiment around the selective rollout of GitHub Copilot (a programming-focused LLM) in October 2021, where GitHub Copilot selectively supported programming languages like Python or Rust, but not R or Haskell. We observe a significant jump in overall contributions, suggesting that LLMs effectively augment collaborative innovation in an unguided setting. Interestingly, Copilot's launch increased iterative innovation focused on maintenance-related or feature-refining contributions significantly more than it did capability innovation through code-development or feature-introducing commits. This disparity was more pronounced after the model upgrade in June 2022 and was evident in active projects with extensive coding activity, suggesting that as both LLM capabilities and/or available contextual information improve, the gap between capability and iterative innovation may widen. We discuss practical and policy implications to incentivize high-value innovative solutions.

研究动机与目标

• 动机：阐明大型语言模型如何影响自愿、主动的开源协作，与组织环境相比。
• 按认知需求区分贡献类型：实质性（新功能）对比 增量式（维护/改进）。
• 利用准实验设计及多重识别策略，识别 Copilot 可用性的因果影响。

提出的方法

• 利用 Copilot 2021 年推出所产生的自然实验，建立一个外生分组（由于业务原因，Python 支持而 R 不支持）。
• 应用三种互补的识别策略来估计因果效应。
• 使用两种分类方法将贡献分类为实质性和增量性。
• 在各种设定中估计总体贡献以及各类型贡献的百分比提升。

实验结果

研究问题

• RQ1 Copilot 的可用性是否会在因果上增加 GitHub 上的开源贡献？
• RQ2 相较于实质性贡献，增量贡献是否受 Copilot 影响更大？
• RQ3 活跃度和模型升级如何调节对现有代码库的开发利用与探索新功能之间的平衡？

主要发现

• Copilot 可用性使开源贡献增加 28–40%。
• 在各规范中，增量贡献的增幅高于实质性贡献。
• 在高活跃度项目中放大效应更大，且在模型升级后扩大。
• 大型语言模型将协作创新倾向于对既有代码库的开发利用，而非对新功能的探索。
• 本研究为快速发展的知识经济中语言模型效应提供因果场域证据。