[Paper Review] A Research Software Engineering Workflow for Computational Science and Engineering

Tomislav Marić, Dennis Gläser+5 more|arXiv (Cornell University)|Aug 15, 2022

Scientific Computing and Data Management22 citations

TL;DR

The paper proposes a lightweight RSE workflow for university-based CSE projects, integrating version control, build systems, cross-linking of code, data, and publications, issue tracking, CI, containerization, TDD, and HPC-based evaluation to improve reproducibility and sustainability of research software.

ABSTRACT

University research groups in Computational Science and Engineering (CSE) generally lack dedicated funding and personnel for Research Software Engineering (RSE), which, combined with the pressure to maximize the number of scientific publications, shifts the focus away from sustainable research software development and reproducible results. The neglect of RSE in CSE at University research groups negatively impacts the scientific output: research data - including research software - related to a CSE publication cannot be found, reproduced, or re-used, different ideas are not combined easily into new ideas, and published methods must very often be re-implemented to be investigated further. This slows down CSE research significantly, resulting in considerable losses in time and, consequentially, public funding. We propose a RSE workflow for Computational Science and Engineering (CSE) that addresses these challenges, that improves the quality of research output in CSE. Our workflow applies established software engineering practices adapted for CSE: software testing, result visualization, and periodical cross-linking of software with reports/publications and data, timed by milestones in the scientific publication process. The workflow introduces minimal work overhead, crucial for university research groups, and delivers modular and tested software linked to publications whose results can easily be reproduced. We define research software quality from a perspective of a pragmatic researcher: the ability to quickly find the publication, data, and software related to a published research idea, quickly reproduce results, understand or re-use a CSE method, and finally extend the method with new research ideas.

Motivation & Objective

• Motivate the need for sustainable research software in university CSE groups and link it to reproducible results and publications.
• Adapt established software engineering practices to CSE with minimal overhead for small teams.
• Define a pragmatic notion of research software quality centered on findability, reproducibility, and reusability.
• Present a lightweight workflow outline that can be extended and integrated with publication milestones.

Proposed method

• Adapts software engineering practices (testing, visualization, cross-linking) to CSE and ties them to milestones in the scientific publication process.
• Proposes a minimal to full workflow built on existing open source tools (VCS, build systems, PID-based cross-linking, CI, containerization).
• Defines a branching model that aligns research ideas with publication milestones and uses Git and tags to capture reproducible states.
• Introduces cross-linking of publications, software, and datasets with persistent identifiers (PIDs) to enhance findability and reproducibility.
• Discusses how HPC-based automatic quantification and visualization can detect performance or numerical quality degradation early.

Experimental results

Research questions

• RQ1 How can a minimal, low-overhead RSE workflow be designed for small university CSE teams?
• RQ2 What tooling and processes enable easy findability, accessibility, interoperability, and reproducibility of research software and results?
• RQ3 How can publication milestones be integrated with software development and data management to support reproducibility?
• RQ4 What is the role of cross-linking (software, data, publications) in sustaining long-term research software in academia?

Key findings

• A minimal workflow using VCS, a build system, cross-linking, and PID-based linking can be adopted by small university groups to improve reproducibility.
• Cross-linking publications, software, and data with PIDs helps ensure findability, accessibility, and reusability of research outputs.
• Automation through CI and testing, including HPC test runs, helps detect degradation in performance or numerical quality over time.
• A publication-centric milestone model provides a practical anchor for integrating testing and integration into research workflows.
• Containerization and ready-made compute environments are beneficial for reproducible experiments across diverse HPC systems.