[Paper Review] The Art and Practice of Data Science Pipelines: A Comprehensive Study of Data Science Pipelines In Theory, In-The-Small, and In-The-Large

Sumon Biswas, Mohammad Wardat+1 more|arXiv (Cornell University)|Dec 2, 2021

Data Quality and Management9 citations

TL;DR

This paper presents a comprehensive study of data science pipelines across three contexts: theoretical frameworks, small-scale implementations (in-the-small), and large-scale production systems (in-the-large). Through analysis of 71 theoretical proposals, 105 Kaggle pipelines, and 21 GitHub projects, the authors identify significant structural differences—particularly the absence of key stages like data storage and deployment in small-scale work—and propose distinct, context-aware representations for each setting, advancing the design and understanding of data science pipelines in practice and research.

ABSTRACT

Increasingly larger number of software systems today are including data science components for descriptive, predictive, and prescriptive analytics. The collection of data science stages from acquisition, to cleaning/curation, to modeling, and so on are referred to as data science pipelines. To facilitate research and practice on data science pipelines, it is essential to understand their nature. What are the typical stages of a data science pipeline? How are they connected? Do the pipelines differ in the theoretical representations and that in the practice? Today we do not fully understand these architectural characteristics of data science pipelines. In this work, we present a three-pronged comprehensive study to answer this for the state-of-the-art, data science in-the-small, and data science in-the-large. Our study analyzes three datasets: a collection of 71 proposals for data science pipelines and related concepts in theory, a collection of over 105 implementations of curated data science pipelines from Kaggle competitions to understand data science in-the-small, and a collection of 21 mature data science projects from GitHub to understand data science in-the-large. Our study has led to three representations of data science pipelines that capture the essence of our subjects in theory, in-the-small, and in-the-large.

Motivation & Objective

• To understand the architectural characteristics of data science pipelines across theoretical frameworks, small-scale implementations, and large-scale production systems.
• To identify discrepancies between theoretical pipeline models and their practical implementations in real-world data science workflows.
• To investigate how pipeline stages, organization, and characteristics vary across different development scales (in-the-small vs. in-the-large).
• To provide actionable, context-aware representations of data science pipelines that inform better design and implementation in research and industry.
• To bridge the gap between theoretical data science lifecycle models and their actual usage in software engineering practice.

Proposed method

• Conducted a systematic analysis of 71 theoretical proposals for data science pipelines from academic literature to extract and categorize common stages and patterns.
• Collected and analyzed 105 data science pipelines from Kaggle competitions to study in-the-small implementations, focusing on single-data-scientist workflows.
• Collected and analyzed 21 mature, team-developed data science projects from GitHub to study in-the-large production systems.
• Used a comparative, multi-context approach to identify recurring stages, organizational structures, and characteristic patterns across the three settings.
• Developed three distinct, context-specific representations of data science pipelines—capturing theory, in-the-small, and in-the-large—based on empirical findings.
• Applied a structured coding scheme to map pipeline stages (e.g., data acquisition, modeling, evaluation) across datasets, enabling cross-context comparison.

Experimental results

Research questions

• RQ1 What are the core stages of data science pipelines in theoretical frameworks, and how do they compare to those in real-world implementations?
• RQ2 How do the organizational structures of data science pipelines differ between in-the-small (Kaggle) and in-the-large (GitHub) contexts?
• RQ3 Which pipeline stages are consistently missing or underrepresented in small-scale data science workflows compared to theoretical models?
• RQ4 What are the dominant patterns and characteristics of data science pipelines in production-grade systems versus academic or competition-based settings?
• RQ5 How do the roles of data preparation, feature engineering, and deployment vary across theoretical, small-scale, and large-scale data science pipelines?

Key findings

• Only six out of eleven theoretical pipeline stages (e.g., data acquisition, preprocessing, modeling, evaluation, deployment, monitoring) were present in in-the-small pipelines, with data storage and monitoring notably absent.
• In-the-small pipelines exhibited a predominantly linear structure with strong emphasis on data exploration and iterative prototyping, lacking the modular and scalable organization seen in in-the-large systems.
• In-the-large projects showed more complex, non-linear pipeline structures with explicit separation of concerns, versioning, and monitoring stages, reflecting production-grade software engineering practices.
• Theoretical frameworks often include stages like data storage and model monitoring, which are frequently omitted in Kaggle-based implementations, indicating a gap between theory and practice.
• Despite theoretical consensus on stages like modeling and evaluation, their implementation frequency and form varied significantly across contexts, with in-the-small pipelines often combining or skipping stages.
• The study revealed that MLOps practices—such as CI/CD, logging, and model versioning—were systematically present in in-the-large projects but largely absent in in-the-small, highlighting a maturity gap in data science engineering.