[Paper Review] Empirical Standards for Software Engineering Research

Paul Ralph, Nauman bin Ali+41 more|arXiv (Cornell University)|Oct 7, 2020

Software Engineering ResearchComputer Science99 citations

TL;DR

This paper proposes living, natural-language empirical standards for common software engineering research methods to improve quality, peer-review fairness, and transparency.

ABSTRACT

Empirical Standards are natural-language models of a scientific community's expectations for a specific kind of study (e.g. a questionnaire survey). The ACM SIGSOFT Paper and Peer Review Quality Initiative generated empirical standards for research methods commonly used in software engineering. These living documents, which should be continuously revised to reflect evolving consensus around research best practices, will improve research quality and make peer review more effective, reliable, transparent and fair.

Motivation & Objective

• Define empirical standards as living, natural-language models for SE research expectations.
• Provide standards for commonly used SE methods to guide research design and reporting.
• Promote transparency, reliability, and fairness in peer review through standardized practices.
• Encourage continuous revision of standards to reflect evolving consensus in the field.

Proposed method

• Adopts the concept of empirical standards as natural-language models of a community's expectations for specific study types.
• Uses the ACM SIGSOFT Paper and Peer Review Quality Initiative as a basis for developing standards.
• Frames standards as living documents to be continuously revised as consensus evolves.

Experimental results

Research questions

• RQ1 What constitutes rigorous standards for common software engineering empirical methods?
• RQ2 How can empirical standards be implemented and revised to remain current and useful for researchers and reviewers?
• RQ3 In what ways can standardized reporting improve transparency and fairness in SE peer review?
• RQ4 How should these standards cover diverse methods (e.g., surveys, experiments, case studies) within SE?

Key findings

• Empirical standards are proposed as living documents to guide research quality and peer review.
• Standards cover commonly used software engineering research methods to improve consistency and fairness in evaluation.
• The initiative aims to make peer review more effective, reliable, transparent, and fair.
• Standards should be continuously revised to reflect evolving consensus in research best practices.