[Paper Review] Smoke Testing for Machine Learning: Simple Tests to Discover Severe Defects

Steffen Herbold, Tobias von der Haar|arXiv (Cornell University)|Sep 3, 2020

Adversarial Robustness in Machine Learning34 references4 citations

TL;DR

This paper proposes a lightweight, generic smoke testing approach for machine learning libraries using equivalence class and boundary value analysis on inputs and hyperparameters. It demonstrates that simple, textbook-based testing techniques can uncover severe, previously unknown bugs—finding 11 in total, including two critical ones—in three mature ML libraries, proving that foundational testing methods remain effective and adaptable for modern ML systems.

ABSTRACT

Machine learning is nowadays a standard technique for data analysis within software applications. Software engineers need quality assurance techniques that are suitable for these new kinds of systems. Within this article, we discuss the question whether standard software testing techniques that have been part of textbooks since decades are also useful for the testing of machine learning software. Concretely, we try to determine generic and simple smoke tests that can be used to assert that basic functions can be executed without crashing. We found that we can derive such tests using techniques similar to equivalence classes and boundary value analysis. Moreover, we found that these concepts can also be applied to hyperparameters, to further improve the quality of the smoke tests. Even though our approach is almost trivial, we were able to find bugs in all three machine learning libraries that we tested and severe bugs in two of the three libraries. This demonstrates that common software testing techniques are still valid in the age of machine learning and that considerations how they can be adapted to this new context can help to find and prevent severe bugs, even in mature machine learning libraries.

Motivation & Objective

• To investigate whether traditional software testing techniques remain effective for modern machine learning systems.
• To identify generic, simple smoke tests that can detect severe defects in ML libraries without requiring complex test oracles.
• To adapt equivalence class and boundary value analysis to inputs and hyperparameters for broader test coverage.
• To evaluate the effectiveness of combinatorial testing strategies on hyperparameters for scalable and effective smoke testing.
• To establish actionable best practices for ML library developers to prevent and detect critical defects early.

Proposed method

• Apply equivalence class analysis to identify problematic input regions, such as extreme values, zero features, and empty or near-empty classes.
• Extend the analysis to hyperparameters by modeling them as test conditions and applying boundary and equivalence class techniques.
• Use a linear-complexity combinatorial strategy that covers each hyperparameter’s equivalence classes once while holding others at default values.
• Design a minimal, reusable smoke test suite targeting core ML functions: model training and prediction.
• Apply the test suite to three mature ML libraries (scikit-learn, Weka, SparkML) to detect real-world bugs.
• Report all findings to maintain transparency and validate the severity and reproducibility of discovered issues.

Experimental results

Research questions

• RQ1 Can classical software testing techniques like equivalence class and boundary value analysis be effectively adapted for machine learning systems?
• RQ2 What are the minimal, generic inputs and hyperparameter configurations that can serve as effective smoke tests for ML libraries?
• RQ3 How can combinatorial testing of hyperparameters be made scalable while ensuring coverage of critical configurations?
• RQ4 To what extent can such simple tests detect severe, previously unknown bugs in mature ML libraries?
• RQ5 What best practices can be derived for developers to integrate these tests into ML library development workflows?

Key findings

• The authors discovered 11 previously unknown bugs in three mature machine learning libraries, including two that were classified as severe.
• All three libraries failed at least one of the proposed smoke tests, indicating widespread issues in core functionality.
• The combinatorial testing approach with linear growth in test cases effectively uncovered bugs that would have been missed by naive or single-parameter testing.
• The proposed smoke test suite successfully detected crashes and incorrect behaviors in training and prediction functions under edge-case inputs.
• The approach is generalizable and effective even in mature systems, suggesting it would be highly beneficial in early-stage ML development.
• All reported bugs were acknowledged by developers, with three already fixed and most others pending patch integration.