[Paper Review] A numerical variability approach to results stability tests and its application to neuroimaging

Yohan Chatelain, Loïc Tetrel+6 more|arXiv (Cornell University)|Jul 3, 2023

Parallel Computing and Optimization TechniquesComputer Science3 citations

TL;DR

This paper proposes a numerical variability approach to results stability testing in neuroimaging, using random rounding of floating-point operations to estimate acceptable variation bounds. Applied to fMRIPrep, the method detects subtle algorithmic changes while accepting inherent numerical variations, significantly enhancing reproducibility in long-term neuroimaging studies.

ABSTRACT

Ensuring the long-term reproducibility of data analyses requires results stability tests to verify that analysis results remain within acceptable variation bounds despite inevitable software updates and hardware evolutions. This paper introduces a numerical variability approach for results stability tests, which determines acceptable variation bounds using random rounding of floating-point calculations. By applying the resulting stability test to \fmriprep, a widely-used neuroimaging tool, we show that the test is sensitive enough to detect subtle updates in image processing methods while remaining specific enough to accept numerical variations within a reference version of the application. This result contributes to enhancing the reliability and reproducibility of data analyses by providing a robust and flexible method for stability testing.

Motivation & Objective

• To address the challenge of ensuring long-term reproducibility in neuroimaging data analysis despite hardware and software evolution.
• To develop a robust, flexible stability test that defines acceptable variation bounds based on inherent numerical variability.
• To apply and validate the method in fMRIPrep, a widely used neuroimaging pre-processing pipeline.
• To ensure the stability test is sensitive enough to detect meaningful algorithmic changes while remaining specific to natural numerical variation.

Proposed method

• Estimate numerical variability using random rounding, a stochastic arithmetic technique that perturbs floating-point results by rounding to adjacent representable values.
• Sample the null distribution of reference results by applying random rounding to the reference application (fMRIPrep) across multiple runs.
• Use a parametric statistical test (Equation 1) to compare the unperturbed result from a new version or environment to the perturbed reference distribution.
• Apply spatial smoothing (FWHM ≥ 15 mm) to approximate normality and improve test specificity, aligning with the statistical assumptions of the test.
• Control for reproducibility by using deterministic execution for the test result and random perturbations only during reference distribution sampling.
• Apply Bonferroni correction for multiple comparisons across voxels, acknowledging the conservative nature of this adjustment due to spatial correlation.

Figure 1: Stability test construction and evaluation. The reference application $\Lambda$ is executed $n$ times on the input data, with random perturbations, resulting in images $X_{k}$ from which a null distribution is built to test the result produced by target application $\tilde{\Lambda}$ . All

Experimental results

Research questions

• RQ1 Can numerical variability estimated via random rounding reliably define acceptable variation bounds for results stability testing in neuroimaging pipelines?
• RQ2 How sensitive is the proposed stability test in detecting meaningful changes in fMRIPrep’s pre-processing algorithms?
• RQ3 To what extent does spatial smoothing improve the specificity and statistical validity of the stability test?
• RQ4 How do variations in hardware, operating system, and parallelization affect the test’s ability to distinguish between numerical noise and actual algorithmic changes?
• RQ5 Can the method be generalized to other scientific domains beyond neuroimaging?

Key findings

• The stability test successfully detected subtle algorithmic changes in fMRIPrep while accepting numerical variations within the reference version, demonstrating high sensitivity and specificity.
• Random rounding produced variability patterns comparable to those from varying random seeds, increasing confidence in the method’s reliability.
• Spatial smoothing with a FWHM of at least 15 mm was necessary to achieve acceptable specificity, as it better aligned the voxel intensity distribution with the normality assumption of the statistical test.
• The test exhibited variable performance across subjects, highlighting the importance of using diverse datasets with varied acquisition parameters to ensure robustness.
• The parametric test with Bonferroni correction was computationally efficient and sufficient for the test’s purpose, though potentially conservative due to voxel correlation.
• The method is generalizable and applicable to other data types, including 4D functional neuroimaging data, with plans to extend the framework in future work.

Figure 2: Voxel-wise means of significant bits measured across n=30 perturbed samples for RR and RS perturbations and 8 subjects.