[Paper Review] A Survey on Spark Ecosystem for Big Data Processing

Shanjiang Tang, Bingsheng He+3 more|arXiv (Cornell University)|Nov 18, 2018

Cloud Computing and Resource Management60 references17 citations

TL;DR

This survey provides a comprehensive, systematic review of optimization techniques for Apache Spark, classifying advancements across six layers of the Spark ecosystem—storage, processing, data management, high-level languages, and applications. It identifies key challenges in performance, generality, and fault tolerance, and outlines future research directions for in-memory big data processing.

ABSTRACT

With the explosive increase of big data in industry and academic fields, it is necessary to apply large-scale data processing systems to analysis Big Data. Arguably, Spark is state of the art in large-scale data computing systems nowadays, due to its good properties including generality, fault tolerance, high performance of in-memory data processing, and scalability. Spark adopts a flexible Resident Distributed Dataset (RDD) programming model with a set of provided transformation and action operators whose operating functions can be customized by users according to their applications. It is originally positioned as a fast and general data processing system. A large body of research efforts have been made to make it more efficient (faster) and general by considering various circumstances since its introduction. In this survey, we aim to have a thorough review of various kinds of optimization techniques on the generality and performance improvement of Spark. We introduce Spark programming model and computing system, discuss the pros and cons of Spark, and have an investigation and classification of various solving techniques in the literature. Moreover, we also introduce various data management and processing systems, machine learning algorithms and applications supported by Spark. Finally, we make a discussion on the open issues and challenges for large-scale in-memory data processing with Spark.

Motivation & Objective

• To systematically review and classify existing research on optimizing Spark for performance and generality.
• To identify limitations in Spark’s current architecture, including memory management, fault tolerance, and hardware support.
• To analyze the evolution of Spark’s ecosystem layers—from low-level storage to high-level applications—and their optimization strategies.
• To highlight open challenges in supporting heterogeneous accelerators (GPU, FPGA, TPU) and fine-grained data operations.
• To guide researchers and practitioners by summarizing state-of-the-art techniques and outlining future research directions in Spark-based big data processing.

Proposed method

• Categorizes Spark ecosystem into six support layers: Storage, Processor, Data Management, Data Processing, High-level Language, and Application Algorithm layers.
• Reviews and classifies optimization techniques in Spark based on their focus: performance (e.g., scheduling, memory I/O), generality (e.g., high-level APIs), and fault tolerance.
• Analyzes Spark’s RDD model, lineage-based recovery, and in-memory computation, identifying bottlenecks such as garbage collection and data skew.
• Evaluates extensions for accelerators (GPU, FPGA, APU, TPU) and their integration challenges due to heterogeneous programming models.
• Examines memory management solutions like Tungsten, which rearchitects Spark’s memory system to reduce GC overhead.
• Reviews Spark-based systems for machine learning (e.g., MLlib), deep learning (e.g., DeepLearning4J, TensorFlow on Spark), and data management (e.g., Shark, Spark SQL).

Experimental results

Research questions

• RQ1 How can Spark’s performance be optimized across different layers of its ecosystem, especially in in-memory data processing?
• RQ2 What are the key limitations of Spark’s RDD model in terms of data skew, immutability, and lack of fine-grained operations?
• RQ3 How can Spark be extended to support emerging heterogeneous accelerators like GPU, FPGA, APU, and TPU?
• RQ4 What are the open challenges in fault tolerance, particularly regarding lineage persistence and driver failure?
• RQ5 What are the most effective optimization strategies for memory management and garbage collection in Spark?

Key findings

• Spark’s in-memory processing provides significantly higher performance than MapReduce, but garbage collection remains a major bottleneck affecting performance.
• The lineage-based fault tolerance mechanism in Spark is more storage-efficient than replication but assumes driver and lineage metadata availability, limiting 100% fault tolerance.
• Tungsten, a Spark memory management project, reduces memory footprint and GC overhead by optimizing serialization and memory layout.
• Data skew in RDD partitions leads to load imbalance, causing performance degradation in pipelined task execution across workers.
• Support for heterogeneous accelerators (GPU, FPGA, TPU) is limited in Spark, requiring new abstractions to unify diverse programming models like CUDA and OpenCL.
• High-level declarative and procedural APIs (e.g., Spark SQL, MLlib) improve usability but do not resolve low-level performance or generality issues in core Spark.