[Paper Review] 9.6 Million Links in Source Code Comments: Purpose, Evolution, and Decay

Hideaki Hata, Christoph Treude+2 more|arXiv (Cornell University)|Jan 22, 2019

Software Engineering Research43 references4 citations

TL;DR

This study analyzes 9.6 million links in source code comments across 25,925 Git repositories to investigate their prevalence, purposes, evolution, and decay. It finds that links are commonly used for licensing, documentation, and attribution, but nearly 10% are dead due to target evolution or lack of updates; link-fixing pull requests were successfully merged in open-source projects, highlighting the fragility of such links and the need for better tooling to maintain them.

ABSTRACT

Links are an essential feature of the World Wide Web, and source code repositories are no exception. However, despite their many undisputed benefits, links can suffer from decay, insufficient versioning, and lack of bidirectional traceability. In this paper, we investigate the role of links contained in source code comments from these perspectives. We conducted a large-scale study of around 9.6 million links to establish their prevalence, and we used a mixed-methods approach to identify the links' targets, purposes, decay, and evolutionary aspects. We found that links are prevalent in source code repositories, that licenses, software homepages, and specifications are common types of link targets, and that links are often included to provide metadata or attribution. Links are rarely updated, but many link targets evolve. Almost 10% of the links included in source code comments are dead. We then submitted a batch of link-fixing pull requests to open source software repositories, resulting in most of our fixes being merged successfully. Our findings indicate that links in source code comments can indeed be fragile, and our work opens up avenues for future work to address these problems.

Motivation & Objective

• To understand the prevalence and purposes of links in source code comments across real-world software repositories.
• To investigate how links and their targets evolve over time and the extent of link decay.
• To evaluate the feasibility and impact of fixing broken links in open-source projects through pull requests.
• To identify opportunities for tool support to improve traceability, versioning, and persistence of external references in code comments.

Proposed method

• Collected 9.6 million links from source code comments in 25,925 Git repositories using a parser supporting 7 programming languages.
• Conducted a large-scale quantitative analysis to assess link prevalence and decay rates across repositories.
• Performed a stratified qualitative study on 1,146 sampled links to classify link targets and purposes.
• Analyzed commit histories to study link evolution and target changes over time.
• Submitted 9 link-fixing pull requests to active open-source projects to evaluate real-world fixability and maintainer responsiveness.
• Combined quantitative and qualitative methods to assess link decay, versioning issues, and bidirectional traceability gaps.

Experimental results

Research questions

• RQ1 How prevalent are links in source code comments across real-world repositories?
• RQ2 What types of targets do links in source code comments typically reference, and what purposes do they serve?
• RQ3 How do links and their targets evolve over time, and to what extent do they suffer from decay?
• RQ4 To what extent can broken links in source code comments be successfully fixed through pull requests in active open-source projects?

Key findings

• Over 80% of the 25,925 studied repositories contained at least one link in source code comments, indicating widespread use.
• Licenses, software homepages, and specifications were the most common types of link targets.
• Links were frequently used to provide metadata or attribution, with 10% of links found to be dead due to target changes or unavailability.
• Despite low update frequency, link targets evolved in many cases, contributing significantly to link decay.
• All nine submitted link-fixing pull requests were successfully merged by project maintainers, demonstrating the feasibility and acceptance of such fixes.
• The study reveals that links in source code comments are fragile due to lack of versioning and bidirectional traceability, posing risks to documentation and software maintenance.