[Paper Review] Shark: SQL and Rich Analytics at Scale

Reynold Xin, Josh Rosen+4 more|arXiv (Cornell University)|Nov 27, 2012

Data Quality and Management19 citations

TL;DR

Shark is a high-performance data analysis system that combines SQL query processing with complex analytics like machine learning using a fault-tolerant in-memory engine. It leverages Resilient Distributed Datasets (RDDs) with columnar storage and dynamic mid-query replanning to achieve up to 100× faster performance than Hive and Hadoop on both SQL and iterative machine learning workloads, while supporting fine-grained fault recovery during long-running queries.

ABSTRACT

Shark is a new data analysis system that marries query processing with complex analytics on large clusters. It leverages a novel distributed memory abstraction to provide a unified engine that can run SQL queries and sophisticated analytics functions (e.g., iterative machine learning) at scale, and efficiently recovers from failures mid-query. This allows Shark to run SQL queries up to 100x faster than Apache Hive, and machine learning programs up to 100x faster than Hadoop. Unlike previous systems, Shark shows that it is possible to achieve these speedups while retaining a MapReduce-like execution engine, and the fine-grained fault tolerance properties that such engines provide. It extends such an engine in several ways, including column-oriented in-memory storage and dynamic mid-query replanning, to effectively execute SQL. The result is a system that matches the speedups reported for MPP analytic databases over MapReduce, while offering fault tolerance properties and complex analytics capabilities that they lack.

Motivation & Objective

• To address the performance bottleneck of MapReduce-based systems for interactive and complex analytics workloads.
• To enable efficient execution of both SQL queries and iterative machine learning algorithms in a unified, fault-tolerant runtime.
• To support fine-grained fault tolerance during long-running queries without requiring full re-execution.
• To combine the speed of MPP databases with the fault tolerance and analytics expressiveness of MapReduce.
• To eliminate the need for data export between systems by enabling native SQL-to-machine learning pipelines.

Proposed method

• Extends the Spark RDD abstraction with in-memory columnar storage and columnar compression to optimize relational data processing.
• Introduces Partial DAG Execution (PDE), a run-time optimization technique that re-plans query execution based on real-time, fine-grained statistics collected during query execution.
• Uses lineage tracking of RDDs to enable deterministic, fine-grained fault recovery by re-computing lost data partitions on other nodes.
• Supports complex analytics by integrating machine learning algorithms natively within the same execution engine as SQL queries.
• Employs a hybrid execution model that combines MapReduce-style task scheduling with database-style optimization techniques for efficient query execution.
• Enables seamless integration with Apache Hive by maintaining compatibility with HiveQL and the Hive metastore.

Experimental results

Research questions

• RQ1 Can a system achieve MPP-database-level performance for SQL queries while retaining the fault tolerance and expressiveness of a MapReduce-like engine?
• RQ2 Can in-memory columnar storage and dynamic query optimization significantly reduce latency in large-scale data processing?
• RQ3 Is it feasible to support both complex analytics and SQL workloads in a single, unified engine with fine-grained fault tolerance?
• RQ4 Can mid-query replanning based on real-time statistics improve performance for long-running, complex queries?
• RQ5 Can such a system deliver 100× speedups over Hadoop and Hive without sacrificing fault tolerance or extensibility?

Key findings

• Shark achieves up to 100× faster query execution than Apache Hive on standard SQL queries, with real-world deployments reporting 40–100× speedups.
• Machine learning workloads, such as logistic regression, run up to 100× faster on Shark than on Hadoop, due to in-memory execution and iterative computation optimization.
• The Partial DAG Execution (PDE) technique enables dynamic query optimization during execution, improving performance by adapting to actual data statistics collected at runtime.
• Shark supports fine-grained fault tolerance by recovering lost data partitions through deterministic lineage re-computation, typically within seconds, even during long-running queries.
• Columnar in-memory storage reduces data size and I/O, contributing to significant performance improvements, especially for analytical workloads with high selectivity and aggregation.
• Shark is the first system to natively combine SQL, machine learning, and fine-grained fault tolerance in a single, open-source engine, eliminating the need for data movement between systems.