[Paper Review] High-Performance Distributed Multi-Model / Multi-Kernel Simulations: A Case-Study in Jungle Computing

Niels Drost, Jason Maassen+7 more|arXiv (Cornell University)|Mar 1, 2012

Distributed and Parallel Computing Systems13 references4 citations

TL;DR

This paper presents a prototype distributed simulation framework built on the Ibis middleware to enable high-performance, multi-model, multi-kernel astrophysical simulations across heterogeneous, geographically dispersed resources—'Jungle Computing' systems. It demonstrates that AMUSE, a computational astrophysics framework, can efficiently utilize remote GPUs across continents, reducing simulation time per iteration from 353 seconds (CPU-only) to 62.4 seconds when distributing models across four institutions.

ABSTRACT

High-performance scientific applications require more and more compute power. The concurrent use of multiple distributed compute resources is vital for making scientific progress. The resulting distributed system, a so-called Jungle Computing System, is both highly heterogeneous and hierarchical, potentially consisting of grids, clouds, stand-alone machines, clusters, desktop grids, mobile devices, and supercomputers, possibly with accelerators such as GPUs. One striking example of applications that can benefit greatly of Jungle Computing Systems are Multi-Model / Multi-Kernel simulations. In these simulations, multiple models, possibly implemented using different techniques and programming models, are coupled into a single simulation of a physical system. Examples include the domain of computational astrophysics and climate modeling. In this paper we investigate the use of Jungle Computing Systems for such Multi-Model / Multi-Kernel simulations. We make use of the software developed in the Ibis project, which addresses many of the problems faced when running applications on Jungle Computing Systems. We create a prototype Jungle-aware version of AMUSE, an astrophysical simulation framework. We show preliminary experiments with the resulting system, using clusters, grids, stand-alone machines, and GPUs.

Motivation & Objective

• To address the challenge of efficiently utilizing highly heterogeneous, distributed, and hierarchical computing resources—termed 'Jungle Computing'—for large-scale scientific simulations.
• To enable multi-model, multi-kernel (3MK) simulations in computational astrophysics that require massive computational power beyond single-system capabilities.
• To develop a prototype system that allows seamless deployment and execution of AMUSE simulations across clusters, grids, stand-alone machines, and GPUs in a distributed environment.
• To evaluate the performance and feasibility of such a system using real-world experiments spanning multiple continents and diverse hardware resources.

Proposed method

• Leveraged the open-source Ibis software framework to abstract low-level system heterogeneity and manage communication and resource access across diverse platforms.
• Extended the AMUSE simulation framework with distributed capabilities to support coupling of multiple astrophysical models (e.g., stellar evolution, hydrodynamics) across different compute resources.
• Implemented a Jungle-aware coupler that dynamically selects and assigns models to appropriate resources based on available hardware, including CPUs and GPUs.
• Used message-passing interfaces (MPI) for inter-node communication and Ibis’s lightweight process model for inter-resource coordination across firewalls and NATs.
• Enabled remote GPU utilization by reassigning compute-intensive models (e.g., Octgrav) to high-performance GPU nodes like the LGM cluster in Leiden.
• Configured the system to allow transparent reconfiguration of simulation components via simple script changes, minimizing user-level complexity.

Experimental results

Research questions

• RQ1 Can a distributed, multi-model, multi-kernel simulation framework be effectively deployed across heterogeneous, geographically dispersed resources in a 'Jungle Computing' environment?
• RQ2 How efficiently can such a system utilize remote GPU resources to accelerate compute-intensive components of a simulation?
• RQ3 What is the performance overhead of managing distributed execution across multiple institutions and network topologies?
• RQ4 Can the system dynamically adapt to resource availability and automatically select optimal compute kernels (e.g., CPU vs. GPU versions) for different models?
• RQ5 To what extent can the system scale performance by distributing models across multiple continents and diverse hardware platforms?

Key findings

• The prototype successfully executed a 3MK simulation across multiple institutions, including a GPU-equipped node in Leiden, demonstrating cross-continental resource utilization.
• Using a remote Tesla C2050 GPU on the LGM cluster reduced the time per simulation iteration to 84 seconds, outperforming a local GeForce 9600GT GPU (89 seconds).
• Distributing the simulation across four different universities reduced the time per iteration to 62.4 seconds, achieving a 5.7x speedup over CPU-only execution (353 seconds).
• The system achieved significant performance gains by offloading compute-intensive models to remote GPUs, even when the network distance was 30 kilometers.
• The overhead of the distributed system was low enough to make remote GPU usage more efficient than local GPU usage in certain configurations.
• The prototype demonstrated that minimal configuration changes (e.g., one line in a script) were sufficient to switch between local and remote execution, proving usability and flexibility.