[Paper Review] Deep Reinforcement Learning achieves flow control of the 2D Karman Vortex Street

Jean Rabault, Ulysse Reglade+3 more|arXiv (Cornell University)|Aug 31, 2018

Model Reduction and Neural Networks17 references3 citations

TL;DR

This paper demonstrates that Deep Reinforcement Learning (DRL) can successfully achieve active flow control of the 2D Kármán vortex street behind a cylinder at Re=100. By training a Deep Artificial Neural Network (DANN) to control synthetic jets, the method reduces drag by approximately 8% and suppresses vortex shedding with minimal actuation, using only 0.35% of the incoming mass flow rate on average in the steady state.

ABSTRACT

The Karman Vortex Street has been investigated for over a century and offers a reference case for investigation of flow stability and control of high dimensionality, non-linear systems. Active flow control, while of considerable interest from a theoretical point of view and for industrial applications, has remained inaccessible due to the difficulty in finding successful control strategies. Here we show that Deep Reinforcement Learning can achieve a stable active control of the Karman vortex street behind a two-dimensional cylinder. Our results show that Deep Reinforcement Learning can be used to design active flow controls and is a promising tool to study high dimensionality, non-linear, time dependent dynamic systems present in a wide range of scientific problems.

Motivation & Objective

• To investigate whether Deep Reinforcement Learning (DRL) can achieve stable active flow control in high-dimensional, nonlinear fluid systems like the 2D Kármán vortex street.
• To evaluate the feasibility of using DRL-trained artificial neural networks to design control strategies for complex fluid flows without relying on reduced-order models.
• To demonstrate that DRL can discover effective control policies with minimal energy input, even in systems with strong nonlinearities and instabilities.
• To explore the potential of DRL as a general-purpose tool for studying and controlling complex dynamical systems in fluid mechanics and beyond.

Proposed method

• A 2D incompressible Navier-Stokes simulation was used to model flow around a cylinder at Re=100, following a standard benchmark configuration.
• Two synthetic jets with angular width 10° were placed on the cylinder sides, injecting fluid normal to the surface with zero net mass flux (Q₁ + Q₂ = 0).
• A Deep Q-Network (DQN) agent was trained using a reward function based on minimizing drag (D) and lift (L) fluctuations to suppress vortex shedding.
• The DRL agent observed the instantaneous flow state and selected control actions (mass flow rates Q₁, Q₂) to maximize cumulative discounted reward.
• The control policy was trained end-to-end using temporal difference learning, with the network updating its weights based on observed state transitions and rewards.
• The method did not rely on reduced-order models or adjoint-based optimization, enabling direct control of the full Navier-Stokes system.

Experimental results

Research questions

• RQ1 Can Deep Reinforcement Learning effectively learn control policies for active flow control in a high-dimensional, nonlinear fluid system like the 2D Kármán vortex street?
• RQ2 To what extent can a DRL agent reduce drag and suppress vortex shedding with minimal actuation energy?
• RQ3 How does the DRL agent’s control strategy alter the mean pressure distribution and recirculation zone behind the cylinder?
• RQ4 Can the DRL agent stabilize a new flow configuration using only small, intermittent control inputs after an initial transient phase?
• RQ5 Is the DRL-based control robust to small perturbations in the control signal, and what does this imply about the sensitivity of the controlled state?

Key findings

• The DRL agent achieved a drag reduction of approximately 8% compared to the baseline uncontrolled flow.
• Vortex shedding fluctuations were nearly suppressed, with the time-resolved drag coefficient (C_D) showing significantly reduced oscillations in the controlled case.
• The control strategy increased the size and altered the pressure distribution in the recirculation bubble behind the cylinder, which is the physical mechanism for drag reduction.
• The DRL agent used very low actuation levels: average normalized mass flow rates were as low as 0.005, amounting to only 0.35% of the incoming mass flow rate in the steady state.
• A two-phase control behavior was observed: a brief initial transient with stronger control (up to ±0.02 normalized flow rate), followed by a pseudo-periodic regime with minimal actuation.
• The system was highly sensitive to control perturbations—slight changes in the control signal caused the collapse of the active control regime, indicating the existence of a narrow, stable operating point.