[Paper Review] deep-significance - Easy and Meaningful Statistical Significance Testing in the Age of Neural Networks

Dennis Ulmer, Christian Hardmeier+1 more|arXiv (Cornell University)|Apr 14, 2022

Explainable Artificial Intelligence (XAI)Computer Science45 references19 citations

TL;DR

This paper introduces deep-significance, an open-source Python package that simplifies statistical significance testing for machine learning and deep learning research. It implements the Almost Stochastic Order (ASO) test and other robust methods to address the underuse of significance testing in ML, demonstrating improved Type I error control and statistical power across diverse data distributions, especially in small-sample and non-normal settings.

ABSTRACT

A lot of Machine Learning (ML) and Deep Learning (DL) research is of an empirical nature. Nevertheless, statistical significance testing (SST) is still not widely used. This endangers true progress, as seeming improvements over a baseline might be statistical flukes, leading follow-up research astray while wasting human and computational resources. Here, we provide an easy-to-use package containing different significance tests and utility functions specifically tailored towards research needs and usability.

Motivation & Objective

• To address the widespread underuse of statistical significance testing (SST) in machine learning and deep learning research, which risks validating spurious improvements.
• To reduce the perceived complexity of SST by providing an easy-to-use, open-source software package tailored for ML practitioners.
• To evaluate and compare the performance of various significance tests—especially ASO—under realistic ML experimental conditions, including small sample sizes and non-normal distributions.
• To provide a comprehensive guide explaining the methods, limitations, and best practices for significance testing in ML research.
• To demonstrate that ASO offers superior Type I error control and statistical power compared to classical tests like t-test and Wilcoxon signed-rank test in typical ML benchmarking scenarios.

Proposed method

• Implements the Almost Stochastic Order (ASO) test, a non-parametric, assumption-free method that compares the cumulative distribution of performance scores between two models.
• Integrates multiple standard significance tests (e.g., Student’s t-test, Wilcoxon signed-rank test, permutation tests, bootstrap resampling) into a unified, user-friendly Python package.
• Employs a simulation-based evaluation framework to assess Type I error rates across various sample sizes and underlying distributions (Gaussian, Laplace, Rayleigh).
• Uses repeated random sampling to estimate statistical power and Type I error rates under controlled conditions, simulating typical ML benchmarking workflows.
• Applies the package to a real-world case study comparing model performance across multiple NLP tasks, validating its practical utility.
• Provides a detailed documentation guide explaining methodological choices, assumptions, and potential pitfalls for researchers.

(a) Stochastic order with red $\succeq$ green.

Experimental results

Research questions

• RQ1 How do different significance tests perform in terms of Type I error control when applied to small, non-normally distributed performance scores common in ML experiments?
• RQ2 Does the Almost Stochastic Order (ASO) test maintain appropriate Type I error rates across diverse data distributions, including heavy-tailed and skewed distributions?
• RQ3 How does ASO compare to classical tests (e.g., t-test, Wilcoxon) in statistical power when detecting true performance differences in small-sample ML settings?
• RQ4 Can a unified, easy-to-use software package significantly increase the adoption of statistical significance testing in ML research?
• RQ5 What are the practical limitations and best practices for applying significance testing in deep learning benchmarking?

Key findings

• ASO maintains near-nominal Type I error rates (close to 0.05) across all tested distributions (Gaussian, Laplace, Rayleigh), even with small sample sizes (n=5), outperforming classical tests that often exceed the nominal threshold.
• Student’s t-test and Wilcoxon signed-rank test exhibit inflated Type I error rates—especially under non-normal distributions—while ASO remains robust.
• Bootstrap and permutation tests show moderate Type I error inflation at small sample sizes, particularly with skewed distributions, whereas ASO remains stable.
• ASO demonstrates higher statistical power than classical tests in detecting true performance differences, especially when distributions are non-normal or sample sizes are small.
• The deep-significance package successfully reduces the barrier to entry for significance testing in ML, enabling researchers to apply robust statistical methods with minimal methodological overhead.
• In the case study, ASO consistently provided reliable inference across multiple NLP tasks, while classical tests yielded inconsistent or misleading results under the same conditions.

(b) Almost stochastic order, with blue $\succsim$ green.