[Paper Review] Multi-agent reinforcement learning for wall modeling in LES of flow over periodic hills

Di Zhou, Michael P. Whitmore+2 more|arXiv (Cornell University)|Nov 29, 2022

Fluid Dynamics and Turbulent Flows4 citations

TL;DR

This paper proposes a multi-agent reinforcement learning (MARL)-based wall model for large-eddy simulation (LES) of turbulent flow over periodic hills, using local flow states to optimize wall eddy viscosity and predict wall-shear stress. The model outperforms traditional equilibrium wall models in capturing mean velocity, pressure, and shear stress fields across varying Reynolds numbers, demonstrating robustness to pressure-gradient effects without requiring high-fidelity training data.

ABSTRACT

We develop a wall model for large-eddy simulation (LES) that takes into account various pressure-gradient effects using multi-agent reinforcement learning (MARL). The model is trained using low-Reynolds-number flow over periodic hills with agents distributed on the wall along the computational grid points. The model utilizes a wall eddy-viscosity formulation as the boundary condition, which is shown to provide better predictions of the mean velocity field, rather than the typical wall-shear stress formulation. Each agent receives states based on local instantaneous flow quantities at an off-wall location, computes a reward based on the estimated wall-shear stress, and provides an action to update the wall eddy viscosity at each time step. The trained wall model is validated in wall-modeled LES (WMLES) of flow over periodic hills at higher Reynolds numbers, and the results show the effectiveness of the model on flow with pressure gradients. The analysis of the trained model indicates that the model is capable of distinguishing between the various pressure gradient regimes present in the flow.

Motivation & Objective

• To develop a data-efficient wall model for LES that adapts to complex pressure-gradient effects in turbulent flows.
• To extend multi-agent reinforcement learning (MARL) from zero-pressure-gradient flows to flows with strong pressure gradients, such as over periodic hills.
• To train a wall model using only local flow states and wall-shear stress recovery as a reward, avoiding reliance on high-fidelity simulation data.
• To validate the model’s performance in wall-modeled LES (WMLES) at higher Reynolds numbers than used in training.
• To assess the model’s ability to generalize across different flow regimes and grid resolutions.

Proposed method

• Distribute multiple reinforcement learning agents at wall-adjacent grid points to learn local wall eddy viscosity adjustments.
• Each agent uses local instantaneous flow quantities (e.g., streamwise velocity, strain rate) as input states and computes actions to update wall eddy viscosity.
• The reward function is based on the accuracy of predicted wall-shear stress relative to reference data, encouraging convergence to correct shear stress.
• The MARL framework trains a shared policy across agents using temporal difference learning, optimizing long-term reward for accurate wall stress prediction.
• The wall model is implemented as a wall eddy-viscosity formulation, replacing standard wall-shear stress boundary conditions in LES.
• Training is performed on low-Reynolds-number periodic hill flow, and the model is tested in higher-Reynolds-number WMLES simulations.

Experimental results

Research questions

• RQ1 Can a MARL-based wall model effectively capture wall-shear stress in flows with strong pressure gradients, such as over periodic hills?
• RQ2 How does the MARL wall model compare to equilibrium wall models (EQWM) in predicting mean velocity and Reynolds stress profiles?
• RQ3 To what extent does the trained MARL model generalize to higher Reynolds numbers not used during training?
• RQ4 Can the MARL model distinguish between different pressure-gradient regimes (e.g., adverse, favorable) in the flow field?
• RQ5 How sensitive is the model’s performance to mesh resolution and subgrid-scale model choice?

Key findings

• The MARL wall model successfully predicts mean wall-shear stress, mean wall pressure, and mean velocity profiles in LES of periodic hills with good accuracy at the training Reynolds number.
• At $Re_H = 10595$, the MARL model outperforms the EQWM in predicting the size and shape of the separation bubble and the peak wall-shear stress on the windward side.
• At $Re_H = 19000$ and $37000$, the model maintains good agreement with WRLES data for mean skin friction coefficient, though performance degrades slightly with increasing Reynolds number.
• The coarse-mesh RLWM simulation at $Re_H = 19000$ shows better agreement with WRLES than the baseline mesh, suggesting potential grid sensitivity.
• The model demonstrates the ability to distinguish between different pressure-gradient regimes, as shown by distinct state-action patterns in regions of adverse and favorable pressure gradients.
• The MARL model achieves improved predictions without requiring high-fidelity training data, confirming its data efficiency and physics-aware learning capability.