[Paper Review] Pilot-Abstraction: A Valid Abstraction for Data-Intensive Applications on HPC, Hadoop and Cloud Infrastructures?

André Luckow, Pradeep Mantha+1 more|arXiv (Cornell University)|Jan 21, 2015

Cloud Computing and Resource Management30 references3 citations

TL;DR

This paper proposes an extended Pilot-Abstraction to unify resource management across HPC, Hadoop, and cloud infrastructures, enabling seamless execution of data-intensive workflows. By integrating YARN and HDFS with the Pilot-Abstraction and introducing Pilot-Data Memory using in-memory backends like Spark, the framework achieves up to 212x speedup for iterative KMeans clustering compared to file-based I/O.

ABSTRACT

HPC environments have traditionally been designed to meet the compute demand of scientific applications and data has only been a second order concern. With science moving toward data-driven discoveries relying more on correlations in data to form scientific hypotheses, the limitations of HPC approaches become apparent: Architectural paradigms such as the separation of storage and compute are not optimal for I/O intensive workloads (e.g. for data preparation, transformation and SQL). While there are many powerful computational and analytical libraries available on HPC (e.g. for scalable linear algebra), they generally lack the usability and variety of analytical libraries found in other environments (e.g. the Apache Hadoop ecosystem). Further, there is a lack of abstractions that unify access to increasingly heterogeneous infrastructure (HPC, Hadoop, clouds) and allow reasoning about performance trade-offs in this complex environment. At the same time, the Hadoop ecosystem is evolving rapidly and has established itself as de-facto standard for data-intensive workloads in industry and is increasingly used to tackle scientific problems. In this paper, we explore paths to interoperability between Hadoop and HPC, examine the differences and challenges, such as the different architectural paradigms and abstractions, and investigate ways to address them. We propose the extension of the Pilot-Abstraction to Hadoop to serve as interoperability layer for allocating and managing resources across different infrastructures. Further, in-memory capabilities have been deployed to enhance the performance of large-scale data analytics (e.g. iterative algorithms) for which the ability to re-use data across iterations is critical. As memory naturally fits in with the Pilot concept of retaining resources for a set of tasks, we propose the extension of the Pilot-Abstraction to in-memory resources.

Motivation & Objective

• Address the growing challenge of managing heterogeneous, data-intensive scientific workflows across HPC, Hadoop, and cloud infrastructures.
• Overcome the limitations of traditional HPC architectures—such as separation of storage and compute—that hinder I/O-intensive workloads.
• Unify access to diverse infrastructures through a common abstraction layer that supports performance-aware workload mapping and resource management.
• Enable interoperability between HPC and Hadoop ecosystems by extending the Pilot-Abstraction to support YARN, HDFS, and in-memory data processing.
• Improve performance for iterative data analytics by introducing Pilot-Data Memory, a pluggable in-memory backend for shared, persistent data across tasks.

Proposed method

• Extend the Pilot-Abstraction to manage Hadoop YARN and HDFS resources, enabling HPC-style resource allocation for data-intensive workloads.
• Implement Pilot-Hadoop to deploy YARN on HPC clusters, allowing Hadoop frameworks to run on compute-centric HPC resources like Stampede.
• Introduce Pilot-Data Memory as a unified abstraction for distributed in-memory data storage, accessible across multiple tasks within a Pilot.
• Support pluggable in-memory backends (e.g., Redis, Spark) via a modular adaptor interface to enable flexible deployment and performance tuning.
• Integrate the Pilot-Abstraction with BigJob for workflow orchestration, enabling end-to-end data pipelines combining HPC and ABDS (Apache Big Data Stack) components.
• Use benchmarking on HPC (Stampede, Gordon) and cloud infrastructures to evaluate performance across file-based and in-memory backends for MapReduce and KMeans workloads.

Experimental results

Research questions

• RQ1 Can the Pilot-Abstraction be effectively extended to support Hadoop YARN and HDFS for interoperability between HPC and data-intensive computing environments?
• RQ2 How does the performance of in-memory data processing via Pilot-Data Memory compare to traditional file-based I/O for iterative data analytics workloads?
• RQ3 What performance improvements can be achieved by using Spark-based in-memory backends versus Redis or file-based storage in data-intensive workflows?
• RQ4 To what extent can the Pilot-Abstraction enable unified, scalable management of heterogeneous infrastructure for complex, multi-stage data workflows?
• RQ5 How does the Pilot-Abstraction support the composition of HPC and ABDS frameworks within a single, cohesive data pipeline?

Key findings

• The Pilot-Abstraction successfully enables interoperability between HPC and Hadoop ecosystems by supporting YARN and HDFS deployment on HPC resources, as demonstrated on Stampede and Gordon.
• Pilot-Data Memory achieves a measured speedup of up to 212x for KMeans clustering compared to the file-based Pilot-Data backend, demonstrating significant performance gains for iterative analytics.
• Spark-based in-memory backends outperform Redis by up to 19x in speedup and show better scale-out efficiency, especially due to Spark’s native distributed architecture.
• The Redis backend achieved only up to 11x speedup, primarily due to its non-distributed deployment, highlighting the importance of native distributed in-memory systems.
• The Pilot-Abstraction enables efficient composition of data-intensive workflows across HPC and ABDS frameworks, supporting stages from data ingestion to advanced analytics.
• The framework’s pluggable backend design allows flexible integration with emerging technologies such as Docker-based containers, ensuring extensibility for evolving infrastructures.