[Paper Review] Multi-Agent Reinforcement Learning for Adaptive Mesh Refinement

Jiachen Yang, Ketan Mittal+6 more|arXiv (Cornell University)|Nov 2, 2022

BIM and Construction Integration4 citations

TL;DR

This paper formulates adaptive mesh refinement (AMR) as a fully-cooperative Markov game where each mesh element acts as an independent agent, using a novel deep multi-agent reinforcement learning algorithm called Value Decomposition Graph Network (VDGN). VDGN enables anticipatory refinement by leveraging graph neural networks and centralized training with decentralized execution, significantly outperforming threshold-based methods in global error and cost metrics while generalizing across diverse mesh geometries and simulation durations.

ABSTRACT

Adaptive mesh refinement (AMR) is necessary for efficient finite element simulations of complex physical phenomenon, as it allocates limited computational budget based on the need for higher or lower resolution, which varies over space and time. We present a novel formulation of AMR as a fully-cooperative Markov game, in which each element is an independent agent who makes refinement and de-refinement choices based on local information. We design a novel deep multi-agent reinforcement learning (MARL) algorithm called Value Decomposition Graph Network (VDGN), which solves the two core challenges that AMR poses for MARL: posthumous credit assignment due to agent creation and deletion, and unstructured observations due to the diversity of mesh geometries. For the first time, we show that MARL enables anticipatory refinement of regions that will encounter complex features at future times, thereby unlocking entirely new regions of the error-cost objective landscape that are inaccessible by traditional methods based on local error estimators. Comprehensive experiments show that VDGN policies significantly outperform error threshold-based policies in global error and cost metrics. We show that learned policies generalize to test problems with physical features, mesh geometries, and longer simulation times that were not seen in training. We also extend VDGN with multi-objective optimization capabilities to find the Pareto front of the tradeoff between cost and error.

Motivation & Objective

• To address the challenge of long-term, anticipatory mesh refinement in time-dependent PDEs where local error estimators fail.
• To overcome the posthumous credit assignment problem in AMR, where agents contributing to future rewards are no longer present at decision time.
• To develop a scalable, generalizable MARL framework for AMR that supports dynamic mesh topology and arbitrary geometries.
• To directly optimize the trade-off between solution error and computational cost using reinforcement learning.
• To demonstrate that MARL can access new regions of the error-cost objective landscape not reachable by traditional threshold-based policies.

Proposed method

• Formulates AMR as a fully-cooperative Markov game, with each mesh element as an independent agent making refinement/de-refinement decisions based on local information.
• Introduces Value Decomposition Graph Network (VDGN), a deep MARL algorithm using graph neural networks to handle unstructured, variable-sized meshes.
• Employs centralized training with decentralized execution (CTDE) to resolve the posthumous credit assignment problem arising from agent creation and deletion.
• Uses graph attention mechanisms to enable long-range neighborhood information aggregation, allowing agents to anticipate future solution features.
• Applies value decomposition to decompose global value functions into local value functions for individual agents, enabling cooperative policy learning.
• Extends VDGN with multi-objective optimization to find the Pareto front between error and computational cost.

Experimental results

Research questions

• RQ1 Can multi-agent reinforcement learning be effectively applied to adaptive mesh refinement to enable anticipatory refinement of future solution features?
• RQ2 How can the posthumous credit assignment problem—caused by agent creation and deletion—be resolved in a MARL framework for AMR?
• RQ3 Can a MARL policy trained on one mesh geometry and PDE feature type generalize to out-of-distribution test cases with different geometries, features, and simulation lengths?
• RQ4 Does MARL outperform traditional threshold-based AMR policies in terms of global error and computational cost?
• RQ5 Can VDGN discover new regions of the error-cost objective landscape not accessible to local error estimator-based methods?

Key findings

• VDGN policies achieve a 1.47x performance improvement over the best threshold-based policy in terms of error-to-cost ratio on 2500-step simulations, compared to 1.10 on shorter episodes, demonstrating superior long-term stability.
• VDGN maintains high performance on test problems with 2500 solver steps, even when numerical error accumulates beyond the training distribution, indicating robustness to long-term error propagation.
• VDGN policies generalize effectively to out-of-distribution test cases, including anisotropic waves, ring-shaped features, opposite-moving waves, and star-shaped meshes, without fine-tuning or retraining.
• A policy trained on quadrilateral elements generalizes rationally to triangular elements, as shown in qualitative results, indicating structural robustness of the learned policy.
• VDGN policies trained on a 16×16 mesh exhibit rational refinement behavior when deployed on a 64×64 mesh, demonstrating scalability and size-agnostic operation.
• VDGN enables anticipatory refinement of regions that will later contain complex features, such as large gradients, which traditional local error-based methods cannot anticipate.