[Paper Review] Towards an OSF-based Registered Report Template for Software Engineering Controlled Experiments

Ana B. M. Bett, Thais Nepomuceno+4 more|arXiv (Cornell University)|Feb 10, 2026

Scientific Computing and Data Management0 citations

TL;DR

This paper proposes an OSF-based Registered Report (RR) Stage-1 template for Software Engineering controlled experiments, analyzes OSF RR types against Jedlitschka 2008 guidelines, and discusses gaps, limitations, and future actions for SE-specific RR guidance.

ABSTRACT

Context: The empirical software engineering (ESE) community has contributed to improving experimentation over the years. However, there is still a lack of rigor in describing controlled experiments, hindering reproducibility and transparency. Registered Reports (RR) have been discussed in the ESE community to address these issues. A RR registers a study's hypotheses, methods, and/or analyses before execution, involving peer review and potential acceptance before data collection. This helps mitigate problematic practices such as p-hacking, publication bias, and inappropriate post hoc analysis. Objective: This paper presents initial results toward establishing an RR template for Software Engineering controlled experiments using the Open Science Framework (OSF). Method: We analyzed templates of selected OSF RR types in light of documentation guidelines for controlled experiments. Results: The observed lack of rigor motivated our investigation of OSF-based RR types. Our analysis showed that, although one of the RR types aligned with many of the documentation suggestions contained in the guidelines, none of them covered the guidelines comprehensively. The study also highlights limitations in OSF RR template customization. Conclusion: Despite progress in ESE, planning and documenting experiments still lack rigor, compromising reproducibility. Adopting OSF-based RRs is proposed. However, no currently available RR type fully satisfies the guidelines. Establishing RR-specific guidelines for SE is deemed essential.

Motivation & Objective

• Improve rigor, transparency, and reproducibility in Software Engineering controlled experiments.
• Explore how OSF RR types align with established SE experimentation guidelines (Jedlitschka et al., 2008).
• Propose an initial OSF-based RR Stage-1 template tailored to SE controlled experiments.
• Identify gaps, limitations, and future actions for OSF-based RR adoption in SE.

Proposed method

• Compare OSF RR types (RR.1, RR.3, RR.10, RR.11) against Jedlitschka 2008 SE guidelines.
• Exclude RR types unsuitable for controlled experiments and map remaining types to guideline sections (documentation, planning, operation, analysis).
• Propose an initial OSF-based RR Stage-1 template for SE controlled experiments.
• Analyze lessons learned, limitations, and implications for future RR development in SE.

Experimental results

Research questions

• RQ1 How can we design an RR template for SE controlled experiments based on existing guidelines?
• RQ2 Which OSF RR types best support the documentation and planning needs of SE controlled experiments?
• RQ3 What gaps remain between OSF RR offerings and Jedlitschka (2008) SE guidelines, and how might they be addressed?
• RQ4 What future actions are needed to advance OSF-based RR adoption in Software Engineering?

Key findings

• OSF RR types do not yet comprehensively satisfy SE guideline coverage; none fully meet Jedlitschka 2008 recommendations.
• RR.3 (Qualitative) maps the most guidelines (33 of 37) compared to RR.1 (31 of 37), indicating higher alignment for qualitative study support.
• RR customization on OSF is limited, preventing a single template that combines RR.1 and RR.3 to cover all guidelines.
• There are notable strengths in OSF-based RR for data-driven, hypothesis-testing studies, but limitations arise for exploratory or design-oriented SE research.
• RRs promote planning, transparency, and early peer input, potentially improving study quality while introducing planning burden and slower phase-1 reviews.
• The study emphasizes the need for SE-specific RR guidelines and notes current OSF RR types insufficiently address all SE experimentation aspects.