[Paper Review] A Literature Survey on Empirical Evidence in Software Engineering

Ekaterina Pek, Ralf Lämmel|arXiv (Cornell University)|Apr 3, 2013

Software Engineering Research18 references3 citations

TL;DR

This paper conducts a grounded theory-based literature survey of 175 long papers from seven top SE conferences (2011–2012) to analyze the use of software corpora in empirical research. It finds that while corpora are nearly ubiquitous, no standard projects or datasets are widely reused, and researchers often tailor corpora to specific needs, suggesting a need for community-curated, domain-specific corpora to improve reproducibility and reuse.

ABSTRACT

Context: Software Engineering research makes use of collections of software artifacts (corpora) to derive empirical evidence from. Goal: To improve quality and reproducibility of research, we need to understand the characteristics of used corpora. Method: For that, we perform a literature survey using grounded theory. We analyze the latest proceedings of seven relevant conferences. Results: While almost all papers use corpora of some kind with the common case of collections of source code of open-source Java projects, there are no frequently used projects or corpora across all the papers. For some conferences we can detect recurrences. We discover several forms of requirements and applied tunings for corpora which indicate more specific needs of research efforts. Conclusion: Our survey feeds into a quantitative basis for discussing the current state of empirical research in software engineering, thereby enabling ultimately improvement of research quality specifically in terms of use (and reuse) of empirical evidence.

Motivation & Objective

• To understand the characteristics and usage patterns of software corpora in empirical software engineering research.
• To identify whether commonly used projects or datasets exist across the broader SE community.
• To investigate how researchers tailor corpora to meet specific research needs.
• To support improved research quality through better reuse and standardization of empirical evidence.

Proposed method

• Conducted a grounded theory-based literature survey on long papers from seven major SE conferences (CSMR, ESEM, ICPC, ICSM, MSR, SCAM, WCRE).
• Analyzed conference proceedings from 2011–2012 using manual coding with iterative validation to identify patterns in corpus use.
• Focused on identifying corpus characteristics such as size, language, evolution, and applied tunings.
• Used a bottom-up coding schema that emerged from the data rather than relying on predefined categories.
• Cross-validated findings through multiple coding passes and constant reference to original papers.
• Excluded journal articles to focus on common research practices reflected in conference proceedings.

Experimental results

Research questions

• RQ1 How frequently do software engineering papers use corpora as collections of empirical evidence?
• RQ2 What are the key characteristics (e.g., size, language, evolution) of the corpora used in SE research?
• RQ3 Are there commonly used projects or datasets across the surveyed papers, or is corpus selection highly idiosyncratic?
• RQ4 To what extent do researchers customize or tune their corpora, and what forms do these tunings take?

Key findings

• 95% of the surveyed long papers (175 out of 184) used corpora, indicating near-universal reliance on empirical evidence.
• The most common corpora are collections of open-source Java projects, though no single project or dataset was reused across the majority of papers.
• Only three conferences showed detectable recurrence of specific projects, with one-fourth of papers in those conferences using the same favored projects.
• Researchers frequently apply custom tunings to corpora—such as filtering by size, age, or functionality—indicating diverse and specific research needs.
• No 'holy grail' corpus exists that satisfies all research needs, and self-collection of corpora remains common due to lack of standardization.
• Despite the diversity, recurring requirements and tuning patterns suggest potential for community-curated, domain-specific corpora to enhance reuse and reproducibility.