[Paper Review] Load Balancing using Hilbert Space-filling Curves for Parallel Reservoir Simulations

Hui Liu, Kun Wang+6 more|arXiv (Cornell University)|Aug 4, 2017

Distributed and Parallel Computing Systems3 citations

TL;DR

This paper proposes a Hilbert space-filling curve (HSFC)-based grid partitioning method for parallel reservoir simulations to achieve efficient load balancing. By mapping 3D structured grids to a 1D Hilbert curve and partitioning the curve into equal intervals, the method ensures spatial locality and balanced workloads across MPI processes. The approach demonstrates excellent scalability and partitioning quality, achieving linear speedup on up to 4,096 MPI processes in large-scale simulations with billions of cells.

ABSTRACT

The goal of load balancing (grid partitioning) is to minimize overall computations and communications, and to make sure that all processors have a similar workload. Geometric methods divide a grid by using a location of a cell while topological methods work with connectivity of cells, which is generally described as a graph. This paper introduces a Hilbert space-filling curve method. A space-filling curve is a continuous curve and defines a map between a one-dimensional space and a multi-dimensional space. A Hilbert space-filling curve is one special space-filling curve discovered by Hilbert and has many useful characteristics, such as good locality, which means that two objects that are close to each other in a multi-dimensional space are also close to each other in a one dimensional space. This property can model communications in grid-based parallel applications. The idea of the Hilbert space-filling curve method is to map a computational domain into a one-dimensional space, partition the one-dimensional space to certain intervals, and assign all cells in a same interval to a MPI. To implement a load balancing method, a mapping kernel is required to convert high-dimensional coordinates to a scalar value and an efficient one-dimensional partitioning module that divides a one-dimensional space and makes sure that all intervals have a similar workload. The Hilbert space-filling curve method is compared with ParMETIS, a famous graph partitioning package. The results show that our Hilbert space-filling curve method has good partition quality. It has been applied to grids with billions of cells, and linear scalability has been obtained on IBM Blue Gene/Q.

Motivation & Objective

• Address the challenge of load imbalance in parallel reservoir simulations, which hinders scalability and performance on large-scale grids.
• Develop a geometric load balancing method that is simpler to implement than graph-based partitioning while maintaining high partitioning quality.
• Enable efficient distribution of structured grid workloads across MPI processes using a space-filling curve to preserve spatial locality.
• Demonstrate linear scalability and high performance on large-scale reservoir simulations using the HSFC method on IBM Blue Gene/Q.
• Provide a practical, scalable alternative to complex graph partitioning tools like ParMETIS for scientific computing workloads.

Proposed method

• Map a 3D structured computational domain into a unit cube to normalize spatial coordinates for Hilbert curve mapping.
• Apply a Hilbert space-filling curve to map 3D cell coordinates to a 1D scalar value, preserving spatial locality through the curve’s continuity and locality-preserving properties.
• Partition the 1D Hilbert curve into Np equal intervals, where Np is the number of MPI processes, assigning all cells in each interval to a single process.
• Use a one-dimensional partitioning module to ensure each MPI process receives a workload as balanced as possible, minimizing load imbalance.
• Implement a mapping kernel to convert 3D grid coordinates into 1D Hilbert values efficiently, enabling fast and scalable partitioning.
• Integrate the HSFC partitioning method into a parallel reservoir simulator to evaluate performance and scalability on real-world reservoir models.

Experimental results

Research questions

• RQ1 Can the Hilbert space-filling curve method achieve load balancing quality comparable to graph-based partitioning methods like ParMETIS in large-scale reservoir simulations?
• RQ2 How does the HSFC-based partitioning method scale across thousands of MPI processes in terms of runtime and communication efficiency?
• RQ3 What is the impact of the partitioning method on the convergence behavior of Newton and linear solvers in reservoir simulation workloads?
• RQ4 How does the performance of the HSFC method compare to ParMETIS in terms of partitioning time, communication volume, and overall simulation time?
• RQ5 To what extent does the HSFC method preserve spatial locality and minimize inter-process communication in structured grid simulations?

Key findings

• The HSFC method achieved linear scalability on IBM Blue Gene/Q using up to 4,096 MPI processes, with simulation time halving as the number of processes doubled.
• For a 5-billion-cell grid, the overall runtime decreased from 1,504.32 seconds (512 processes) to 193.52 seconds (4,096 processes), demonstrating strong scalability.
• The Newton method and linear solver showed consistent convergence behavior across all process counts, indicating stable and balanced computation distribution.
• Partitioning time scaled nearly linearly, though it was slightly worse than ideal due to a single MPI process handling the one-dimensional partitioning step.
• The gridding, linear system assembly, and solution phases exhibited excellent scalability, with the GMRES(30) solver and RAS preconditioner maintaining high efficiency.
• The HSFC method produced partitioning quality comparable to ParMETIS, making it a viable and simpler alternative to complex graph partitioning tools for large-scale reservoir simulations.