[Paper Review] ForestClaw: Hybrid forest-of-octrees AMR for hyperbolic conservation laws

Carsten Burstedde, Donna Calhoun+2 more|arXiv (Cornell University)|Aug 7, 2013

Computational Fluid Dynamics and Aerodynamics17 references18 citations

TL;DR

ForestClaw introduces a hybrid forest-of-octrees adaptive mesh refinement (AMR) framework that couples p4est's scalable tree-based AMR with Clawpack's patch-based solvers for hyperbolic conservation laws. By treating each AMR leaf as a uniform computational patch with $m^d$ degrees of freedom, it enables high-performance, load-balanced simulations on up to 16,384 CPU cores with 82% parallel efficiency at scale.

ABSTRACT

We present a new hybrid paradigm for parallel adaptive mesh refinement (AMR) that combines the scalability and lightweight architecture of tree-based AMR with the computational efficiency of patch-based solvers for hyperbolic conservation laws. The key idea is to interpret each leaf of the AMR hierarchy as one uniform compute patch in $\sR^d$ with $m^d$ degrees of freedom, where $m$ is customarily between 8 and 32. Thus, computation on each patch can be optimized for speed, while we inherit the flexibility of adaptive meshes. In our work we choose to integrate with the p4est AMR library since it allows us to compose the mesh from multiple mapped octrees and enables the cubed sphere and other nontrivial multiblock geometries. We describe aspects of the parallel implementation and close with scalings for both MPI-only and OpenMP/MPI hybrid runs, where the largest MPI run executes on 16,384 CPU cores.

Motivation & Objective

• To develop a scalable, high-performance AMR framework for hyperbolic conservation laws that combines the flexibility of tree-based AMR with the efficiency of patch-based solvers.
• To enable efficient parallelism on modern manycore architectures by integrating MPI and OpenMP threading models within a modular, composable design.
• To support complex multiblock geometries such as the cubed sphere by leveraging p4est's block-structured mesh composition capabilities.
• To minimize communication overhead in AMR by optimizing neighbor exchange patterns and ghost cell management across patch boundaries.
• To achieve high parallel efficiency and strong scaling on large-scale HPC systems, including up to 16,384 CPU cores in hybrid MPI/OpenMP execution.

Proposed method

• Maps each p4est octree leaf to a single computational patch in $\mathbb{R}^d$ with $m^d$ degrees of freedom, enabling optimized kernel performance per patch.
• Uses p4est as the underlying AMR infrastructure to manage mesh refinement, coarsening, and parallel partitioning with minimal metadata overhead.
• Integrates with Clawpack/Manyclaw to perform time integration using the wave propagation algorithm on each patch, maintaining numerical accuracy and stability.
• Implements a hybrid MPI+OpenMP parallel model where each MPI rank manages local patches and uses OpenMP threads for intra-node parallelism on each patch.
• Propagates neighbor patch connectivity information from p4est to ForestClaw via a constant-time lookup interface, enabling efficient ghost cell exchange across non-trivial block boundaries.
• Enforces the 2:1 refinement balance condition across levels using p4est’s parallel algorithms to maintain mesh consistency and avoid non-local refinement artifacts.

Experimental results

Research questions

• RQ1 Can a hybrid forest-of-octrees AMR framework achieve high scalability and performance for hyperbolic conservation laws by combining tree-based mesh management with patch-based solvers?
• RQ2 How does the patch-based approach on AMR leaves compare to traditional cell-by-cell finite volume methods in terms of computational efficiency and parallel scaling?
• RQ3 What is the impact of adaptive mesh refinement with subcycling on overall simulation runtime and load balance in large-scale simulations?
• RQ4 How efficiently can ghost cell exchanges be optimized in a multiblock, non-uniform AMR environment with complex block connectivity?
• RQ5 To what extent can the integration of p4est and Clawpack enable support for complex geometries like the cubed sphere while maintaining high parallel efficiency?

Key findings

• The largest MPI-only run achieved 82% parallel efficiency on 16,384 CPU cores, demonstrating strong weak scalability up to 4,096 cores with 96% efficiency.
• Adaptive mesh refinement with subcycling reduced total wall clock time by up to a factor of 50 compared to uniformly refined meshes, with runtimes dropping from 252 seconds (P=16) to 17.3 seconds (P=256).
• Ghost patch exchanges accounted for less than 16% of total time in weak scaling runs up to 4,096 cores, indicating manageable communication overhead.
• The method achieved linear scaling of time integration per step across 16 to 256 MPI ranks, with step times decreasing from 1.115s to 0.092s, confirming good load balance.
• The hybrid MPI+OpenMP model enabled strong scaling on 16,384 cores with 82% efficiency, and the spherical advection example showed stable refinement alignment with the front across partitions.
• The 2:1 balance condition was enforced via p4est’s parallel algorithms, preventing non-local refinement effects and ensuring mesh consistency across levels.