[Paper Review] Enabling Loosely-Coupled Serial Job Execution on the IBM BlueGene/P Supercomputer and the SiCortex SC5832

Ioan Raicu, Zhao Zhang+2 more|ArXiv.org|Aug 26, 2008

Distributed and Parallel Computing Systems25 references3 citations

TL;DR

This paper presents a framework enabling efficient execution of loosely-coupled serial jobs on large-scale supercomputers like IBM BlueGene/P and SiCortex SC5832 without modifying applications. By optimizing I/O and task scheduling, it achieves high scalability—up to 4096 processors on BlueGene/P and 5832 on SiCortex—with sustained execution rates of thousands of tasks per second, demonstrating effectiveness across diverse domains including energy modeling and molecular dynamics.

ABSTRACT

Our work addresses the enabling of the execution of highly parallel computations composed of loosely coupled serial jobs with no modifications to the respective applications, on large-scale systems. This approach allows new-and potentially far larger-classes of application to leverage systems such as the IBM Blue Gene/P supercomputer and similar emerging petascale architectures. We present here the challenges of I/O performance encountered in making this model practical, and show results using both micro-benchmarks and real applications on two large-scale systems, the BG/P and the SiCortex SC5832. Our preliminary benchmarks show that we can scale to 4096 processors on the Blue Gene/P and 5832 processors on the SiCortex with high efficiency, and can achieve thousands of tasks/sec sustained execution rates for parallel workloads of ordinary serial applications. We measured applications from two domains, economic energy modeling and molecular dynamics.

Motivation & Objective

• Address the challenge of efficiently executing large numbers of independent serial jobs on petascale supercomputers.
• Overcome I/O performance bottlenecks that hinder scalability in loosely-coupled job execution models.
• Enable existing serial applications to run efficiently on large-scale systems without modification.
• Demonstrate high scalability and performance on two distinct petascale architectures: IBM BlueGene/P and SiCortex SC5832.
• Validate the approach using real-world applications from diverse domains such as economic energy modeling and molecular dynamics.

Proposed method

• Design a runtime system that schedules and manages independent serial jobs across thousands of processors.
• Implement optimized I/O strategies to reduce contention and improve throughput in large-scale job execution.
• Use lightweight process management and job dispatching to minimize overhead in task spawning and coordination.
• Leverage the underlying hardware’s low-latency interconnect and high-density compute nodes for efficient job distribution.
• Integrate with existing job management systems to ensure compatibility and seamless deployment on production supercomputers.
• Apply load balancing and fault tolerance mechanisms to maintain high utilization and reliability during long-running workloads.

Experimental results

Research questions

• RQ1 How can loosely-coupled serial jobs be efficiently scheduled and executed on petascale systems like BlueGene/P and SiCortex SC5832?
• RQ2 What I/O performance bottlenecks arise in large-scale job execution, and how can they be mitigated?
• RQ3 To what extent can existing serial applications be executed at scale without modification on modern supercomputers?
• RQ4 What is the maximum achievable task throughput and scalability on these architectures using the proposed approach?
• RQ5 How does the performance of the framework vary across different application domains such as molecular dynamics and energy modeling?

Key findings

• The system achieved sustained execution rates of thousands of tasks per second on both the IBM BlueGene/P and SiCortex SC5832.
• Scalability was demonstrated up to 4096 processors on BlueGene/P and 5832 processors on SiCortex SC5832 with high efficiency.
• I/O optimization significantly reduced bottlenecks, enabling high-throughput job execution at scale.
• Real-world applications from economic energy modeling and molecular dynamics showed strong performance and scalability.
• The framework enabled execution of unmodified serial applications on large-scale systems, expanding the class of usable workloads.
• Micro-benchmarks and production applications confirmed the framework’s effectiveness in achieving high parallel efficiency.