[Paper Review] GRACOS: Scalable and Load Balanced P3M Cosmological N-body Code

Alexander Shirokov, Edmund Bertschinger|arXiv (Cornell University)|May 5, 2005

Scientific Research and Discoveries3 references3 citations

TL;DR

GRACOS is a scalable, load-balanced P3M cosmological N-body code for distributed memory clusters that combines a static 1D slab decomposition for long-range forces (Fourier-based mesh) with a dynamic Hilbert curve-based domain decomposition for short-range forces (direct summation). It achieves near-optimal load balance and strong scalability up to 80 processors in $800^3$ simulations, demonstrating effectiveness even under extreme mass inhomogeneity.

ABSTRACT

We present a parallel implementation of the particle-particle/particle-mesh (P3M) algorithm for distributed memory clusters. The GRACOS (GRAvitational COSmology) code uses a hybrid method for both computation and domain decomposition. Long-range forces are computed using a Fourier transform gravity solver on a regular mesh; the mesh is distributed across parallel processes using a static one-dimensional slab domain decomposition. Short-range forces are computed by direct summation of close pairs; particles are distributed using a dynamic domain decomposition based on a space-filling Hilbert curve. A nearly-optimal method was devised to dynamically repartition the particle distribution so as to maintain load balance even for extremely inhomogeneous mass distributions. Tests using $800^3$ simulations on a 40-processor beowulf cluster showed good load balance and scalability up to 80 processes. We discuss the limits on scalability imposed by communication and extreme clustering and suggest how they may be removed by extending our algorithm to include adaptive mesh refinement.

Motivation & Objective

• To develop a message-passing P3M N-body code for distributed memory clusters, specifically Beowulf clusters, to overcome limitations of existing P3M implementations.
• To address the challenge of load imbalance in N-body simulations with highly inhomogeneous mass distributions.
• To achieve strong scalability and near-optimal load balance in cosmological simulations using a hybrid domain decomposition strategy.
• To enable efficient large-scale cosmological simulations with high dynamic range by combining Fourier-based long-range forces with direct short-range force computation.

Proposed method

• Uses a static one-dimensional slab decomposition for distributing the particle-mesh (PM) force computation across processes, enabling efficient FFT-based long-range force calculation.
• Employs a dynamic domain decomposition based on the Hilbert space-filling curve to distribute particles for direct short-range force computation, preserving spatial locality.
• Dynamically repartitions particles using Hilbert curve indices to maintain load balance during simulation, even as particle clustering evolves.
• Maps particle positions to Hilbert curve indices to enable efficient, locality-preserving domain decomposition and reduce communication overhead.
• Utilizes the FFTW library for distributed FFTs on the mesh, with each process handling a slab of the mesh to minimize communication.
• Avoids repeated expensive Hilbert curve function calls by organizing memory access patterns to favor direct array indexing over repeated function invocations.

Experimental results

Research questions

• RQ1 How can a P3M N-body code be efficiently parallelized on distributed memory Beowulf clusters to maintain load balance under extreme particle clustering?
• RQ2 What is the optimal dynamic domain decomposition strategy for minimizing load imbalance in cosmological N-body simulations with evolving inhomogeneous particle distributions?
• RQ3 Can a hybrid approach combining static mesh decomposition and dynamic particle decomposition achieve strong scalability in large-scale cosmological simulations?
• RQ4 How does the performance of the Hilbert curve-based particle distribution compare to other space-filling curves in terms of load balance and communication efficiency?
• RQ5 What are the communication and scalability limits of the current P3M algorithm, and how can they be mitigated using adaptive mesh refinement?

Key findings

• GRACOS achieved good load balance and strong scalability up to 80 processors in $800^3$ simulations on a 40-node Beowulf cluster.
• The dynamic Hilbert curve-based particle decomposition effectively maintained load balance even under extreme mass clustering, minimizing idle time across processors.
• The code demonstrated nearly optimal load balancing by dynamically repartitioning particles based on their evolving spatial distribution.
• The Hilbert curve mapping enabled efficient, locality-preserving domain decomposition, reducing communication costs and improving cache performance.
• The CPU time for a single Hilbert curve function call (hilbert_c2i) at $m=9$ was approximately 1.056 nanoseconds, which is about 120 times slower than a triple array dereference, but performance was optimized by minimizing repeated calls.
• The hybrid method combining static mesh decomposition and dynamic particle decomposition successfully balanced computational load while maintaining high scalability and efficiency.