[論文レビュー] Machine Learning Testing: Survey, Landscapes and Horizons

Jie M. Zhang, Mark Harman+2 more|arXiv (Cornell University)|Jun 19, 2019

Adversarial Robustness in Machine Learning参考文献 235被引用数 108

ひとこと要約

this survey defines ML testing, presents a comprehensive review of 144 ML testing papers across properties, components, workflows, and applications, and outlines trends and future directions.

ABSTRACT

This paper provides a comprehensive survey of Machine Learning Testing (ML testing) research. It covers 144 papers on testing properties (e.g., correctness, robustness, and fairness), testing components (e.g., the data, learning program, and framework), testing workflow (e.g., test generation and test evaluation), and application scenarios (e.g., autonomous driving, machine translation). The paper also analyses trends concerning datasets, research trends, and research focus, concluding with research challenges and promising research directions in ML testing.

研究の動機と目的

• Define Machine Learning Testing (ML testing) and clarify its scope and terminology.
• Survey 144 ML testing papers across properties, components, workflows, and application scenarios.
• Analyze distributions, datasets, and trends in ML testing literature.
• Identify open problems, challenges, and promising directions for ML testing research.

提案手法

• Organize literature along four dimensions: testing properties (e.g., correctness, robustness, fairness), ML components (data, learning program, framework), testing workflow (test generation, execution, evaluation), and application scenarios (e.g., autonomous driving, machine translation).
• Conduct quantitative and qualitative analysis of collected papers including distribution across ML categories and datasets.
• Highlight differences between ML testing and traditional software testing to clarify unique challenges (e.g., data-driven behavior, Oracle problem).
• Synthesize horizons by outlining challenges and potential research directions in ML testing.

実験結果

リサーチクエスチョン

• RQ1 What definitions and scope best capture Machine Learning Testing (ML testing) and its relation to software testing?
• RQ2 What is the landscape of ML testing literature in terms of properties, components, workflows, and applications?
• RQ3 What trends, datasets, and distribution patterns characterize ML testing research?
• RQ4 What are the main challenges and promising directions for future ML testing research?

主な発見

• 85% of ML testing papers appeared since 2016, signaling a rapid rise in interest.
• Approximately 120 papers tackle supervised learning testing, 3 tackle unsupervised learning testing, and 1 addresses reinforcement learning.
• Most studies (93) focus on correctness and robustness, with fewer papers targeting interpretability, privacy, or efficiency.
• The survey identifies a range of testing properties, components, and workflows, and contrasts ML testing with traditional software testing.
• ML testing faces unique challenges such as data quality, emergent system behavior, and the Oracle problem in ML contexts.