[Paper Review] A Survey of Pipeline Tools for Data Engineering

Anthony Mbata, Somayajulu Sripada+1 more|arXiv (Cornell University)|Jun 12, 2024

Algorithms and Data Compression4 citations

TL;DR

This survey evaluates pipeline tools for data engineering, categorizing them into ETL/ELT, data integration, orchestration, and machine learning pipelines. It demonstrates their use on the IDEAL Household Energy Dataset, concluding that Apache Airflow offers superior scalability, extensibility, and workflow management for complex data engineering workflows despite a steep initial learning curve.

ABSTRACT

Currently, a variety of pipeline tools are available for use in data engineering. Data scientists can use these tools to resolve data wrangling issues associated with data and accomplish some data engineering tasks from data ingestion through data preparation to utilization as input for machine learning (ML). Some of these tools have essential built-in components or can be combined with other tools to perform desired data engineering operations. While some tools are wholly or partly commercial, several open-source tools are available to perform expert-level data engineering tasks. This survey examines the broad categories and examples of pipeline tools based on their design and data engineering intentions. These categories are Extract Transform Load/Extract Load Transform (ETL/ELT), pipelines for Data Integration, Ingestion, and Transformation, Data Pipeline Orchestration and Workflow Management, and Machine Learning Pipelines. The survey also provides a broad outline of the utilization with examples within these broad groups and finally, a discussion is presented with case studies indicating the usage of pipeline tools for data engineering. The studies present some first-user application experiences with sample data, some complexities of the applied pipeline, and a summary note of approaches to using these tools to prepare data for machine learning.

Motivation & Objective

• To identify and review existing pipeline tools that address data engineering challenges in data science and AI projects.
• To categorize pipeline tools based on design, functionality, and use cases such as ETL/ELT, data integration, orchestration, and ML pipelines.
• To evaluate the practical usability and performance of selected tools using real-world data, particularly the IDEAL Household Energy Dataset.
• To provide recommendations for tool selection based on scalability, extensibility, and integration capabilities.
• To guide researchers and practitioners in selecting appropriate pipeline tools for data preparation and machine learning deployment.

Proposed method

• Categorization of pipeline tools into four main groups: ETL/ELT, data integration/ingestion/transformation, orchestration and workflow management, and machine learning pipelines.
• Evaluation of tools including Apache Spark (PySpark), Apache Airflow, and TensorFlow Extended (TFX) using the IDEAL Household Energy Dataset.
• Implementation of data processing workflows in Jupyter notebooks and Airflow DAGs to assess functionality and usability.
• Use of TensorFlow Data Validation (TFDV) within TFX to detect data anomalies and monitor data quality.
• Leveraging Airflow’s Python-based DAGs, hooks, and operators to orchestrate multi-tool workflows, including Spark and TFX jobs.
• Comparison of setup complexity, extensibility, and monitoring capabilities across tools, with emphasis on Airflow’s configurability and production readiness.

Figure 1: An ideal pipeline for data engineering (adapted from [ 5 ] ). The schematic points to data sourcing from multiple sources whether sourced in batches or streaming data. The data processing aspects cover the activities that occur between receiving the data and outputting the same for further

Experimental results

Research questions

• RQ1 Which pipeline tools are most effective for handling data engineering tasks such as ingestion, transformation, and integration?
• RQ2 How do different pipeline tools compare in terms of usability, extensibility, and support for machine learning workflows?
• RQ3 What are the key challenges in setting up and maintaining data engineering pipelines, particularly for real-world datasets like the IDEAL Household Energy Dataset?
• RQ4 How do orchestration tools like Apache Airflow improve workflow management and scalability in data engineering pipelines?
• RQ5 What role do data validation and quality monitoring tools play in ensuring reliable machine learning input?

Key findings

• Apache Airflow emerged as the preferred tool due to its high scalability, strong extensibility via Python-based hooks and operators, and robust workflow management capabilities.
• Initial setup of Apache Airflow is complex, particularly on Windows or without Docker, but usability improves significantly after configuration.
• PySpark in Jupyter notebooks enabled fast prototyping and data processing, but lacked native orchestration and monitoring features.
• TFX provided end-to-end machine learning pipeline support, with TFDV enabling automated data quality checks and anomaly detection.
• Airflow’s ability to integrate with Spark, TFX, and other tools through plug-ins allowed for unified, scalable data engineering workflows.
• The survey demonstrated that combining tools like Airflow with PySpark and TFX enables reproducible, testable, and production-ready data pipelines.

Figure 2: An indication of some data organisation, data integration and feature engineering capabilities of Apache Spark. In A the initial reading of the household sensor csv data with spark.read.csv shows the absence of feature names. In B StructType and StructField were used to create dataframe sc