[Paper Review] Discussion Paper: Should statistics rescue mathematical modelling?

Andrea Saltelli|arXiv (Cornell University)|Dec 18, 2017

Probabilistic and Robust Engineering Design132 references3 citations

TL;DR

This paper argues that statistics should play a central role in rescuing mathematical modelling from methodological, cultural, and ethical shortcomings by integrating statistical tools like uncertainty quantification, sensitivity analysis, and sensitivity auditing into modelling practices. The key contribution is a call to standardize and institutionalize these techniques across disciplines to improve model quality and reproducibility.

ABSTRACT

Statistics experiences a storm around the perceived misuse and possible abuse of its methods in the context of the so-called reproducibility crisis. The methods and styles of quantification practiced in mathematical modelling rarely make it to the headlines, though modelling practitioners writing in disciplinary journals flag a host of problems in the field. Technical, cultural and ethical dimensions are simultaneously at play in the current predicaments of both statistics and mathematical modelling. Since mathematical modelling is not a discipline like statistics, its shortcomings risk remaining untreated longer. We suggest that the tools of statistics and its disciplinary organisation might offer a remedial contribution to mathematical modelling, standardising methodologies and disseminating good practices. Statistics could provide scientists and engineers from all disciplines with a point of anchorage for sound modelling work. This is a vast and long-term undertaking. A step in the proposed direction is offered here by focusing on the use of statistical tools for quality assurance of mathematical models. By way of illustration, techniques for uncertainty quantification, sensitivity analysis and sensitivity auditing are suggested for incorporation in statistical syllabuses and practices.

Motivation & Objective

• To address the growing crisis in reproducibility and methodological rigor in scientific modelling, particularly in non-statistical disciplines.
• To highlight that mathematical modelling, unlike statistics, lacks institutionalized standards and is vulnerable to misuse due to its informal practices.
• To propose that statistics can serve as a disciplinary anchor for sound modelling by providing methodological tools and quality assurance frameworks.
• To advocate for embedding statistical techniques such as sensitivity analysis and uncertainty quantification into modelling curricula and practices.
• To promote ethical and technical improvements in modelling through statistical oversight and standardization.

Proposed method

• Proposes the use of uncertainty quantification techniques to assess the reliability of model outputs under input variability.
• Introduces sensitivity analysis as a method to identify which model inputs most influence the output, enhancing transparency and robustness.
• Advocates for sensitivity auditing as a quality control mechanism to evaluate the validity and consistency of sensitivity analysis results.
• Recommends integrating these statistical tools into formal statistical syllabuses and interdisciplinary modelling training programs.
• Draws on existing statistical methodologies to provide a structured, repeatable approach to model validation and error detection.
• Uses the O'Neill conjecture and recent debates on retiring significance testing as illustrative examples of broader methodological challenges in science.

Experimental results

Research questions

• RQ1 How can statistical methodologies improve the reliability and transparency of mathematical models in scientific and engineering applications?
• RQ2 Why has mathematical modelling remained vulnerable to methodological flaws despite the existence of robust statistical tools?
• RQ3 What role can statistics play in standardizing and institutionalizing best practices in mathematical modelling across disciplines?
• RQ4 How can uncertainty quantification and sensitivity analysis be systematically applied to enhance model quality and reproducibility?
• RQ5 What are the ethical and cultural barriers to adopting statistical practices in mathematical modelling, and how can they be overcome?

Key findings

• Statistical tools such as uncertainty quantification and sensitivity analysis can significantly enhance the quality and defensibility of mathematical models.
• The integration of sensitivity auditing into modelling workflows can detect and correct flawed sensitivity analysis practices, improving model integrity.
• The paper identifies a gap in institutional oversight for mathematical modelling, which lacks the formal standards found in statistics.
• The O'Neill conjecture is used as a case study to illustrate how statistical reasoning can clarify and validate complex modelling assumptions.
• Recent debates on retiring significance testing highlight the need for a broader methodological reform that includes mathematical modelling.
• There is a strong argument for including statistical quality assurance techniques in modelling education and professional practice to ensure reproducibility and rigor.