[Paper Review] Are Large Language Models a Threat to Digital Public Goods? Evidence from Activity on Stack Overflow

Maria del Rio-Chanona, Nadzeya Laurentsyeva+1 more|arXiv (Cornell University)|Jul 14, 2023

Expert finding and Q&A systems23 citations

TL;DR

The paper shows that after ChatGPT’s release, Stack Overflow postings declined by about 16% (rising to ~25% over six months) relative to counterfactual platforms, with no significant change in post voting and greater declines for more popular languages.

ABSTRACT

Large language models like ChatGPT efficiently provide users with information about various topics, presenting a potential substitute for searching the web and asking people for help online. But since users interact privately with the model, these models may drastically reduce the amount of publicly available human-generated data and knowledge resources. This substitution can present a significant problem in securing training data for future models. In this work, we investigate how the release of ChatGPT changed human-generated open data on the web by analyzing the activity on Stack Overflow, the leading online Q\&A platform for computer programming. We find that relative to its Russian and Chinese counterparts, where access to ChatGPT is limited, and to similar forums for mathematics, where ChatGPT is less capable, activity on Stack Overflow significantly decreased. A difference-in-differences model estimates a 16\% decrease in weekly posts on Stack Overflow. This effect increases in magnitude over time, and is larger for posts related to the most widely used programming languages. Posts made after ChatGPT get similar voting scores than before, suggesting that ChatGPT is not merely displacing duplicate or low-quality content. These results suggest that more users are adopting large language models to answer questions and they are better substitutes for Stack Overflow for languages for which they have more training data. Using models like ChatGPT may be more efficient for solving certain programming problems, but its widespread adoption and the resulting shift away from public exchange on the web will limit the open data people and models can learn from in the future.

Motivation & Objective

• Assess whether LLMs like ChatGPT substitute human-generated open data on Q&A platforms.
• Quantify the change in Stack Overflow posting activity after ChatGPT’s release using a difference-in-differences design.
• Analyze whether the shift affects content quality via voting data.
• Explore heterogeneity of impact across programming languages and relate it to language popularity on GitHub.

Proposed method

• Use a difference-in-differences model comparing Stack Overflow with four counterfactual platforms (Math Stack Exchange, Math Overflow, Russian Stack Overflow, Segmentfault).
• Model weekly posts with IHS transformation to interpret effects as percent changes; include platform fixed effects, week fixed effects, and platform-specific trends.
• Estimate the post-ChatGPT effect via the interaction of treated=Stack Overflow and post-ChatGPT period; test pretrends with week-specific interactions.
• Complement with an event-study design to examine language-level heterogeneity across 69 language-tagged topics.
• Analyze voting data (upvotes/downvotes) as a proxy for post quality before and after ChatGPT release.
• Correlate estimated language-level effects with GitHub language popularity and developer salary data.

Figure 1 : A) Time series of weekly posts to Stack Overflow since early 2016. The number of weekly posts decreases at a rate of about 7,000 posts each year from 2016 to 2022. In the six months after the release of ChatGPT, the weekly posting rate decreases by around 20,000 posts. B) Comparing posts

Experimental results

Research questions

• RQ1 Does the release of ChatGPT reduce Stack Overflow posting activity relative to comparable, less-affected platforms?
• RQ2 Is ChatGPT displacing high-quality or low-quality content, as measured by voting activity?
• RQ3 Do the effects of ChatGPT differ across programming languages, and are these differences related to language popularity or market signals?

Key findings

• Stack Overflow posting activity decreased by about 15.6% after ChatGPT’s release, rising to roughly 25% within six months.
• Voting activity (upvotes/downvotes) remained stable, suggesting post quality did not decline on average.
• Heterogeneous effects across languages: more widely used languages (e.g., Python, JavaScript) showed larger declines in posting activity.
• Languages with more GitHub repositories tended to experience larger negative effects on Stack Overflow posting after ChatGPT’s release.
• Results are robust to alternative specifications and subsamples (e.g., only questions, weekday posts).

Figure 2 : Difference-in-differences analysis for posting activities. The dashed line marks November 30, 2023 the release date of ChatGPT. Eight weeks after its introduction, we observe a steady decline in the activity of Stack Overflow. The plotted coefficients correspond to the interaction between