[Paper Review] A Data Quality-Driven View of MLOps

Cédric Renggli, Luka Rimanić+4 more|arXiv (Cornell University)|Feb 15, 2021

Machine Learning and Data Classification49 references43 citations

TL;DR

The paper analyzes how data quality dimensions propagate through MLOps stages and proposes principled methods (CPClean, BER estimation, CI, ModelPicker) to align data quality with downstream ML performance.

ABSTRACT

Developing machine learning models can be seen as a process similar to the one established for traditional software development. A key difference between the two lies in the strong dependency between the quality of a machine learning model and the quality of the data used to train or perform evaluations. In this work, we demonstrate how different aspects of data quality propagate through various stages of machine learning development. By performing a joint analysis of the impact of well-known data quality dimensions and the downstream machine learning process, we show that different components of a typical MLOps pipeline can be efficiently designed, providing both a technical and theoretical perspective.

Motivation & Objective

• Highlight the dependency between ML model quality and underlying data quality dimensions (accuracy, completeness, consistency, timeliness).
• Present a unified data-quality perspective across stages of an MLOps lifecycle.
• Showcase four case studies (data cleaning, feasibility study, CI/CD, model selection) that connect data quality to ML outcomes.

Proposed method

• Model noise propagation from data quality to ML training using a framework like CPClean to minimize training data cleaning via conditional entropy of possible data worlds.
• Propose BER-based feasibility study methods using non-parametric, hyper-parameter-free estimators with pre-trained embeddings (ease.ml/snoopy).
• Introduce CI/CD testing for ML with probabilistic test conditions and a sample-size budget accounting for multiple evaluations (ease.ml/ci).
• Present ModelPicker, an online model selection method that queries labels selectively to rank pre-trained models with no-regret guarantees on adversarial streams.

Experimental results

Research questions

• RQ1 How do accuracy, completeness, consistency, and timeliness of data propagate through the ML training and evaluation pipeline?
• RQ2 Can we design data-quality aware MLOps components that improve ML quality without excessive human labeling?
• RQ3 How can feasibility (BER) be estimated and used to prevent unrealistic ML project expectations without expensive training?
• RQ4 How to ensure robust continuous testing and model selection under label-budget constraints in production ML systems?
• RQ5 What are the theoretical limits and practical algorithms for data-aware model selection and cleaning in MLOps?

Key findings

• Data quality dimensions strongly influence ML model quality, and addressing data quality is as important as tuning models.
• CPClean provides a principled way to model noise propagation and guide cleaning by minimizing entropy of possible data worlds, with efficient implementations for kNN classifiers.
• BER estimation can be performed without hyper-parameters using non-parametric, embedding-based nearest-neighbor methods to enable practical feasibility studies.
• CI/CD for ML introduces probabilistic test outcomes requiring confidence-bound based evaluation and optimized test-set re-use to prevent overfitting to test data.
• ModelPicker offers a no-regret online model selection strategy under adversarial and stochastic streams, reducing labeling costs in production data scenarios.