[Paper Review] Quality and sustainability of software tools in neuroscience

Marc-Oliver Gewaltig, Robert C. Cannon|arXiv (Cornell University)|May 14, 2012

Scientific Computing and Data Management20 references3 citations

TL;DR

This paper evaluates the quality and sustainability of 36 neuroscience software tools, identifying critical gaps in maturity, documentation, and maintainability. It proposes development models and end-product classifications to guide researchers, reviewers, and developers in improving software reliability and research readiness through actionable checklists.

ABSTRACT

In computational neuroscience, the continual need for new software tools has led to the creation of software in a disparate and ad-hoc fashion with considerable overlap but little compatibility between different tools. Much of the resulting software is made freely available, but it is not always clear that it is sufficiently mature in terms of domain coverage, validity, documentation or usability, to be useful to other researchers. Software tools for scientific use should meet certain minimal conditions of correctness, usability and reliability. For databases and tools that are made publicly available, additional criteria of maintainability and sustainability should be met. We focused on 36 neuroscience software tools and asked how the culture and practice of software development affects the validity and trustworthiness of scientific software. We identified two sets of categories: one for how software is developed, and one for the end products of such work. The development models include pet projects driven by single individuals or small groups, collaborative projects, and projects that are driven with up-front funding. The end products can be classified as proof of concept work, private tools and public tools according to their accessibility and state of readiness for scientific work. Based on our findings we suggest ways in which the current practice of software development in computational neuroscience can be improved along with checklists for developers, reviewers, and scientists to help assess the quality and research readiness of a particular piece of software.

Motivation & Objective

• To investigate how software development practices affect the validity and trustworthiness of scientific software in computational neuroscience.
• To identify common shortcomings in software maturity, including documentation, usability, and maintainability.
• To classify software tools into development models (pet projects, collaborative, funded) and end-product types (proof of concept, private, public) for better assessment.
• To provide practical checklists for developers, reviewers, and scientists to evaluate software quality and research readiness.
• To improve the sustainability and reliability of neuroscience software through structured development and evaluation practices.

Proposed method

• Analyzed 36 neuroscience software tools across different development models and end-product categories.
• Classified tools based on accessibility and readiness: proof of concept, private tools, and public tools.
• Evaluated software using criteria for correctness, usability, reliability, and maintainability.
• Identified recurring issues in documentation, domain coverage, and compatibility across tools.
• Proposed checklists tailored for developers, reviewers, and scientists to assess software quality.
• Suggested improvements in software development culture to enhance long-term sustainability and scientific trustworthiness.

Experimental results

Research questions

• RQ1 How do different software development models (e.g., individual projects, funded initiatives) affect the quality and sustainability of neuroscience software?
• RQ2 To what extent do publicly available neuroscience software tools meet minimal standards for correctness, usability, and reliability?
• RQ3 What are the key gaps in documentation, compatibility, and maintainability across existing neuroscience software tools?
• RQ4 How can software be better classified by maturity and accessibility to improve research readiness?
• RQ5 What practical checklists can developers, reviewers, and scientists use to assess software quality and sustainability?

Key findings

• Many neuroscience software tools are created in an ad-hoc manner with little compatibility or overlap management, leading to redundancy and inefficiency.
• A significant proportion of publicly available tools lack sufficient documentation, usability support, or maturity for reliable scientific use.
• Development models such as pet projects often result in tools that are not sustainable or maintainable over time.
• Public tools frequently fail to meet criteria for long-term maintainability and reliability, despite being freely available.
• The proposed checklists for developers, reviewers, and scientists provide a structured framework to assess software quality and research readiness.
• Improved software development practices and standardized evaluation criteria can significantly enhance the trustworthiness and sustainability of neuroscience software.