[Paper Review] Intermediate Data Caching Optimization for Multi-Stage and Parallel Big Data Frameworks

Zhengyu Yang, Danlin Jia+3 more|arXiv (Cornell University)|Apr 27, 2018

Caching and Content Delivery34 references3 citations

TL;DR

This paper proposes an adaptive caching algorithm for multi-stage big data frameworks like Apache Spark that automatically identifies and caches the most valuable intermediate datasets (RDDs) across jobs to minimize total recomputation work. By modeling the problem as a DAG-based optimization and using online decision-making with optimality guarantees, the method reduces total work by 12% compared to LRU and other heuristics, especially under increasing cache sizes.

ABSTRACT

In the era of big data and cloud computing, large amounts of data are generated from user applications and need to be processed in the datacenter. Data-parallel computing frameworks, such as Apache Spark, are widely used to perform such data processing at scale. Specifically, Spark leverages distributed memory to cache the intermediate results, represented as Resilient Distributed Datasets (RDDs). This gives Spark an advantage over other parallel frameworks for implementations of iterative machine learning and data mining algorithms, by avoiding repeated computation or hard disk accesses to retrieve RDDs. By default, caching decisions are left at the programmer's discretion, and the LRU policy is used for evicting RDDs when the cache is full. However, when the objective is to minimize total work, LRU is woefully inadequate, leading to arbitrarily suboptimal caching decisions. In this paper, we design an algorithm for multi-stage big data processing platforms to adaptively determine and cache the most valuable intermediate datasets that can be reused in the future. Our solution automates the decision of which RDDs to cache: this amounts to identifying nodes in a direct acyclic graph (DAG) representing computations whose outputs should persist in the memory. Our experiment results show that our proposed cache optimization solution can improve the performance of machine learning applications on Spark decreasing the total work to recompute RDDs by 12%.

Motivation & Objective

• Address the suboptimal caching decisions in Spark caused by manual RDD caching and LRU eviction, which lead to high recomputation costs.
• Formulate the intermediate data caching problem as an optimization task to minimize total work in multi-stage, parallel big data processing.
• Design an online, adaptive caching algorithm that detects cross-job computational overlaps and selects the most valuable RDDs for caching.
• Improve cache utilization and performance by leveraging global job-level access patterns rather than per-job heuristics.
• Validate the approach in real Spark deployments and simulations across diverse workloads, including machine learning and data mining.

Proposed method

• Model the computation as a Directed Acyclic Graph (DAG) where nodes represent operations and edges represent data dependencies.
• Define a mathematical optimization framework to select which intermediate RDDs to cache to minimize total recomputation cost.
• Develop an adaptive caching algorithm that makes online decisions based on future reuse predictions and access patterns across multiple jobs.
• Use a global view of all accessed RDDs to detect cross-job overlaps and prioritize caching of frequently reused or high-cost-to-recompute datasets.
• Implement the algorithm in Apache Spark and compare it with standard policies (FIFO, LRU, LCS) using real workloads and synthetic stress tests.
• Apply optimality guarantees to ensure the algorithm performs near-optimally under the defined objective function.

Experimental results

Research questions

• RQ1 How can intermediate data caching be automated in multi-stage big data frameworks to reduce total recomputation work?
• RQ2 What is the impact of cross-job computational overlaps on cache efficiency, and how can they be exploited effectively?
• RQ3 Can an adaptive caching algorithm outperform traditional heuristics like LRU and LCS in terms of hit ratio and makespan?
• RQ4 How does the performance of the proposed algorithm scale with increasing cache size and workload complexity?
• RQ5 What is the trade-off between cache utilization and computational cost when optimizing for total work minimization?

Key findings

• The proposed adaptive caching algorithm reduces the total work required to recompute RDDs by 12% compared to LRU and other baseline policies.
• The algorithm improves hit ratio by up to 13% and reduces makespan by up to 12% under a cache-unfriendly stress test with limited performance headroom.
• Performance gains are most significant as cache size increases, demonstrating the algorithm's ability to effectively utilize available memory resources.
• Unlike FIFO, LRU, and LCS, the adaptive algorithm detects and exploits cross-job computational overlaps, leading to better long-term caching decisions.
• The algorithm maintains high efficiency even under complex, real-world workloads involving machine learning and data mining pipelines.
• The results show that global, job-level analysis of intermediate data reuse leads to significantly better cache utilization than per-job heuristics.