[Paper Review] An Explorative Study of GitHub Repositories of AI Papers

Boyang Zhang|arXiv (Cornell University)|Feb 16, 2019

Scientific Computing and Data Management7 references4 citations

TL;DR

This exploratory study analyzes over 1,700 GitHub repositories linked to AI papers, revealing widespread issues such as poor code quality, lack of documentation, and insufficient maintenance. The authors identify that these repositories often become inactive and unusable, hindering reproducibility and widening the academia-industry gap, and propose recommendations to improve code sharing practices in AI research.

ABSTRACT

With the rapid development of AI technologies, thousands of AI papers are being published each year. Many of these papers have released sample code to facilitate follow-up researchers. This paper presents an explorative study of over 1700 code repositories of AI papers hosted on GitHub. We find that these repositories are often poorly written, lack of documents, lack of maintenance, and hard to configure the underlying runtime environment. Thus, many code repositories become inactive and abandoned. Such a situation makes follow-up researchers hard to reproduce the results or do further research. In addition, these hard-to-reuse code makes a gap between academia and industry. Based on the findings, we give some recommendations on how to improve the quality of code repositories of AI papers.

Motivation & Objective

• To investigate the quality and sustainability of code repositories associated with AI papers on GitHub.
• To identify common shortcomings in repository maintenance, documentation, and code structure.
• To understand how these issues impede reproducibility and hinder follow-up research.
• To bridge the gap between academic research and industrial practice by improving code sharing standards.

Proposed method

• The study analyzed 1,700 GitHub repositories linked to AI papers published in major conferences.
• Repositories were evaluated based on code quality, documentation completeness, maintenance activity, and ease of environment setup.
• The analysis included checking for README files, code structure, dependency management, and commit frequency.
• Qualitative assessment was used to classify repositories by maintainability and usability.
• The study examined metadata such as repository age, contributor count, and issue tracking activity.
• Findings were synthesized into actionable recommendations for researchers and institutions.

Experimental results

Research questions

• RQ1 How well-maintained are the code repositories associated with AI papers on GitHub?
• RQ2 To what extent do these repositories provide adequate documentation and setup instructions?
• RQ3 What are the primary barriers preventing researchers from reproducing or extending the work in these repositories?
• RQ4 How do the quality and sustainability of these repositories compare across different AI subfields or publication venues?
• RQ5 What systemic improvements can be made to enhance code reuse and reproducibility in AI research?

Key findings

• Over 1,700 AI paper repositories were analyzed, revealing widespread issues in code quality and maintainability.
• A significant proportion of repositories lack essential documentation, making them difficult to understand and use.
• Many repositories are inactive, with no recent commits, indicating abandonment after publication.
• Runtime environment setup is often complex and poorly documented, deterring reproduction efforts.
• The lack of maintainability and usability creates a substantial barrier to reproducibility in AI research.
• These shortcomings contribute to a growing gap between academic research and industrial application of AI technologies.