[论文解读] Are Large Language Models a Threat to Digital Public Goods? Evidence from Activity on Stack Overflow

Maria del Rio-Chanona, Nadzeya Laurentsyeva+1 more|arXiv (Cornell University)|Jul 14, 2023

Expert finding and Q&A systems被引用 23

一句话总结

本论文显示，在 ChatGPT 发布之后，相对对照平台，Stack Overflow 的帖子量下降约16%（六个月内上升至约25%），投票后评分并无显著变化，对于更受欢迎的语言下降更明显。

ABSTRACT

Large language models like ChatGPT efficiently provide users with information about various topics, presenting a potential substitute for searching the web and asking people for help online. But since users interact privately with the model, these models may drastically reduce the amount of publicly available human-generated data and knowledge resources. This substitution can present a significant problem in securing training data for future models. In this work, we investigate how the release of ChatGPT changed human-generated open data on the web by analyzing the activity on Stack Overflow, the leading online Q\&A platform for computer programming. We find that relative to its Russian and Chinese counterparts, where access to ChatGPT is limited, and to similar forums for mathematics, where ChatGPT is less capable, activity on Stack Overflow significantly decreased. A difference-in-differences model estimates a 16\% decrease in weekly posts on Stack Overflow. This effect increases in magnitude over time, and is larger for posts related to the most widely used programming languages. Posts made after ChatGPT get similar voting scores than before, suggesting that ChatGPT is not merely displacing duplicate or low-quality content. These results suggest that more users are adopting large language models to answer questions and they are better substitutes for Stack Overflow for languages for which they have more training data. Using models like ChatGPT may be more efficient for solving certain programming problems, but its widespread adoption and the resulting shift away from public exchange on the web will limit the open data people and models can learn from in the future.

研究动机与目标

• 评估如 ChatGPT 这样的LLM 是否替代了问答平台上人类生成的开放数据。
• 使用差分中的差分设计量化 ChatGPT 发布后 Stack Overflow 的发帖活动变化。
• 分析该转变是否通过投票数据影响内容质量。
• 探索对编程语言的影响异质性，并将其与 GitHub 上的语言受欢迎程度联系起来。

提出的方法

• 使用差分中的差分模型，将 Stack Overflow 与四个对照平台（Math Stack Exchange、Math Overflow、Russian Stack Overflow、Segmentfault）进行比较。
• 对每周发帖量进行 IHS 转换建模，以将效应解释为百分比变化；包含平台固定效应、周固定效应以及平台特定趋势。
• 通过 treated=Stack Overflow 与 post-ChatGPT 期间的交互项来估计 ChatGPT 之后的效应；使用周特定交互检验事前趋势。
• 辅以事件研究设计，考察 69 个语言标签主题的语言层面异质性。
• 分析投票数据（赞成票/反对票）作为发帖质量的代理，在 ChatGPT 发布前后进行比较。
• 将估计的语言层面效应与 GitHub 语言流行度和开发者薪资数据相关联。

Figure 1 : A) Time series of weekly posts to Stack Overflow since early 2016. The number of weekly posts decreases at a rate of about 7,000 posts each year from 2016 to 2022. In the six months after the release of ChatGPT, the weekly posting rate decreases by around 20,000 posts. B) Comparing posts

实验结果

研究问题

• RQ1 ChatGPT 的发布是否相对于较少受影响的可比平台降低了 Stack Overflow 的发帖活动？
• RQ2 是否通过投票活动来衡量内容质量的高低受到 ChatGPT 的取代影响？
• RQ3 ChatGPT 的效应是否因编程语言而异，这些差异是否与语言的流行度或市场信号有关？

主要发现

• ChatGPT 发布后 Stack Overflow 的发帖活动约下降 15.6%，六个月内上升至约 25%。
• 投票活动（赞成票/反对票）保持稳定，表明总体内容质量未下降。
• 语言间存在异质性效应：使用更广泛的语言（如 Python、JavaScript）显示更大幅度的发帖下降。
• 在更多 GitHub 仓库的语言，在 ChatGPT 发布后对 Stack Overflow 发帖的负面影响更大。
• 结果对替代规格和子样本(例如仅问题、工作日发帖)具有稳健性。

Figure 2 : Difference-in-differences analysis for posting activities. The dashed line marks November 30, 2023 the release date of ChatGPT. Eight weeks after its introduction, we observe a steady decline in the activity of Stack Overflow. The plotted coefficients correspond to the interaction between