[Paper Review] Code Review For and By Scientists

Marian Petre, Greg Wilson|arXiv (Cornell University)|Jul 21, 2014

Software Engineering Research9 references15 citations

TL;DR

This paper investigates code review practices for scientists through two pilot studies, finding that just-in-time, small-scale code reviews are more effective than end-of-project reviews. Scientists benefit from incremental, collaborative code review with mentors, improving code quality, reproducibility, and reusability—especially when paired with regular dialogue and clear goals.

ABSTRACT

We describe two pilot studies of code review by and for scientists. Our principal findings are that scientists are enthusiastic, but need to be shown code review in action, and that just-in-time review of small code changes is more likely to succeed than large-scale end-of-work reviews.

Motivation & Objective

• To explore the feasibility and benefits of code review for scientists who typically lack formal software engineering practices.
• To identify barriers to adopting code review in scientific software development, including documentation, tooling, and communication gaps.
• To develop practical, scalable strategies for introducing code review into scientific workflows through mentorship and iterative practice.
• To understand how developers and scientists perceive code review differently and how to bridge these differences for effective collaboration.
• To establish guidelines for implementing sustainable, low-overhead code review processes in scientific research teams.

Proposed method

• Conducted post-hoc code reviews of scientific software from PLOS Computational Biology using professional developers unfamiliar with the scientific domain.
• Performed semi-structured interviews with 11 developers and 4 code authors to assess review quality, usability, and perceived value.
• Launched a second pilot study pairing experienced scientific programmers with less-experienced scientists to transfer code review skills through mentorship.
• Emphasized synchronous, regular review sessions (every 3–4 weeks) with defined goals, schedules, and mutual feedback to build rapport and trust.
• Encouraged incremental improvements—such as writing docstrings and tests with each code change—rather than large-scale refactoring.
• Used real-world codebases from active research projects, focusing on self-contained, 3–4 page modules under active development for initial review.

Experimental results

Research questions

• RQ1 Can non-scientist developers effectively review scientific software when they lack domain expertise?
• RQ2 How do scientists and developers differ in their expectations, standards, and practices regarding code quality and documentation?
• RQ3 What are the key barriers to adopting code review in scientific software development, and how can they be overcome?
• RQ4 How does just-in-time, small-scale code review compare to large-scale, end-of-project reviews in terms of effectiveness and adoption?
• RQ5 What role does interpersonal communication and mentorship play in successful code review adoption among scientists?

Key findings

• Scientists are enthusiastic about code review but often lack experience with software engineering practices such as version control, testing, and documentation.
• Post-hoc, asynchronous code reviews by non-scientist developers were perceived as 'drive-by' due to lack of context, limited access to code, and absence of dialogue.
• Reviewers struggled to validate code without working examples or build instructions, and many had to make assumptions due to poor documentation.
• Scientists often do not package or document their code for reuse, reflecting a cultural norm where code quality is undervalued compared to scientific results.
• Both developers and scientists desired more dialogue during code review, as it enhances understanding, improves feedback quality, and fosters trust.
• The most effective code review processes involved regular, synchronous conversations, clear goals, and incremental improvements—such as adding docstrings and tests with each change—rather than large-scale overhauls.