[Paper Review] Production Machine Learning Pipelines: Empirical Analysis and Optimization Opportunities

Doris Xin, Hui Miao+2 more|arXiv (Cornell University)|Mar 30, 2021

Scientific Computing and Data ManagementDecision Sciences44 references17 citations

TL;DR

This paper presents a large-scale empirical analysis of 3,000 production ML pipelines at Google, revealing widespread inefficiencies from redundant computation in non-deployed models. By introducing 'model graphlets' and a proactive detection system using random forest models on pipeline provenance, the authors reduce wasted computation by 50% without sacrificing model freshness, demonstrating significant optimization opportunities via data management techniques.

ABSTRACT

Machine learning (ML) is now commonplace, powering data-driven applications in various organizations. Unlike the traditional perception of ML in research, ML production pipelines are complex, with many interlocking analytical components beyond training, whose sub-parts are often run multiple times on overlapping subsets of data. However, there is a lack of quantitative evidence regarding the lifespan, architecture, frequency, and complexity of these pipelines to understand how data management research can be used to make them more efficient, effective, robust, and reproducible. To that end, we analyze the provenance graphs of 3000 production ML pipelines at Google, comprising over 450,000 models trained, spanning a period of over four months, in an effort to understand the complexity and challenges underlying production ML. Our analysis reveals the characteristics, components, and topologies of typical industry-strength ML pipelines at various granularities. Along the way, we introduce a specialized data model for representing and reasoning about repeatedly run components in these ML pipelines, which we call model graphlets. We identify several rich opportunities for optimization, leveraging traditional data management ideas. We show how targeting even one of these opportunities, i.e., identifying and pruning wasted computation that does not translate to model deployment, can reduce wasted computation cost by 50% without compromising the model deployment cadence.

Motivation & Objective

• To understand the architectural complexity, operational cadence, and computational characteristics of real-world production ML pipelines beyond model training.
• To identify and quantify inefficiencies in ML pipelines, particularly redundant computation that does not lead to model deployment.
• To explore optimization opportunities using data management principles such as incremental view maintenance, provenance tracking, and materialization.
• To evaluate the effectiveness of machine learning models in predicting which trained models are not deployed, enabling proactive pruning of wasted computation.
• To demonstrate that significant cost savings can be achieved without compromising model freshness or deployment cadence.

Proposed method

• The authors analyze provenance graphs from 3,000 TFX pipelines over four months, involving over 450,000 trained models, to extract pipeline topology and execution patterns.
• They introduce 'model graphlets'—a specialized data model to represent and reason about repeatedly executed components in ML pipelines, capturing input, preprocessing, and training stages.
• A machine learning model (Random Forest) is trained on features including input data statistics, preprocessing steps, and code change patterns to predict whether a trained model will be deployed.
• The system uses a decision function based on predicted deployment likelihood to proactively prune non-deployed models, reducing wasted computation while preserving model freshness.
• The approach is evaluated using a real-world dataset, with key metrics including model freshness and total wasted compute cost.
• Feature ablation studies are conducted to assess the relative importance of different input features, such as input data, preprocessing, and code changes.

Experimental results

Research questions

• RQ1 What are the typical architectural topologies and component types in large-scale production ML pipelines?
• RQ2 How much overlap exists between repeated executions of pipeline components, and how do data distributions and model outputs evolve across runs?
• RQ3 What proportion of trained models in production pipelines are never deployed, and what causes this wasted computation?
• RQ4 Can machine learning models trained on provenance and execution metadata effectively predict which models will not be deployed, enabling proactive optimization?
• RQ5 To what extent can optimization techniques from data management—such as incremental computation and materialization—reduce wasted resources without affecting model freshness?

Key findings

• The analysis reveals that 50% of all wasted computation in production ML pipelines stems from models that are trained but never deployed to downstream services.
• By proactively identifying and pruning such non-deployed models using a Random Forest model trained on pipeline provenance, the system reduces wasted computation by 50% without affecting model freshness.
• Model freshness remains at 100% when pruning up to 50% of wasted computation, but drops sharply beyond 60%, indicating a critical threshold for cost-saving trade-offs.
• The input data features alone (RF:Input) are the most effective predictor of model deployment, outperforming more complex feature sets including code changes and operator shapes.
• Adding code change features to input data features (RF:History) provides no performance improvement, indicating that code changes are uncorrelated with model deployment decisions.
• The feature ablation study shows no single feature group dominates performance, suggesting complex, non-linear interactions between different pipeline characteristics drive prediction accuracy.