[Paper Review] Exploiting locality and physical invariants to design effective Deep Reinforcement Learning control of the unstable falling liquid film

Vincent Belus, Jean Rabault+4 more|arXiv (Cornell University)|Oct 17, 2019

Fluid Dynamics and Turbulent Flows5 references4 citations

TL;DR

This paper presents a novel Deep Reinforcement Learning (DRL) approach for controlling unstable interfacial waves in a 1D falling liquid film by exploiting translational invariance and spatial locality. By designing a convolutional DRL agent with a dense, localized reward function, the method efficiently controls an arbitrary number of actuators, overcoming the curse of dimensionality in high-output control spaces and enabling fast, stable learning with superior performance compared to non-invariant or sparse-reward baselines.

ABSTRACT

Instabilities arise in a number of flow configurations. One such manifestation is the development of interfacial waves in multiphase flows, such as those observed in the falling liquid film problem. Controlling the development of such instabilities is a problem of both academic and industrial interest. However, this has proven challenging in most cases due to the strong nonlinearity and high dimensionality of the underlying equations. In the present work, we successfully apply Deep Reinforcement Learning (DRL) for the control of the one-dimensional (1D) depth-integrated falling liquid film. In addition, we introduce for the first time translational invariance in the architecture of the DRL agent, and we exploit locality of the control problem to define a dense reward function. This allows to both speed up learning considerably, and to easily control an arbitrary large number of jets and overcome the curse of dimensionality on the control output size that would take place using a naive approach. This illustrates the importance of the architecture of the agent for successful DRL control, and we believe this will be an important element in the effective application of DRL to large two-dimensional (2D) or three-dimensional (3D) systems featuring translational, axisymmetric or other invariants.

Motivation & Objective

• To develop an effective DRL control strategy for the unstable falling liquid film, a highly nonlinear and high-dimensional fluid dynamics problem.
• To address the curse of dimensionality in DRL control when managing many actuators by leveraging physical invariants and spatial locality.
• To improve learning efficiency and policy quality by designing a dense, local reward function that provides fine-grained feedback.
• To demonstrate that architectural design—specifically translational invariance and locality—significantly enhances DRL performance in physical systems.
• To enable scalable, generalizable DRL control applicable to complex 2D and 3D fluid systems with similar symmetries and spatial structures.

Proposed method

• Introduces a convolutional DRL agent architecture that explicitly enforces translational invariance, mimicking CNNs in image processing to generalize across spatial positions.
• Designs a dense, local reward function that evaluates control performance at each actuator location, providing fine-grained feedback instead of global averaging.
• Employs a 1D depth-integrated model of the falling liquid film with a semi-implicit finite volume scheme for accurate and efficient simulation.
• Uses Proximal Policy Optimization (PPO) with a custom OpenAI Gym environment to train the DRL agent in a continuous control setting.
• Implements multiple control strategies: M1 (global reward), M2 (sparse reward), and M3 (dense, local reward) to compare learning efficiency and policy quality.
• Applies environment splitting and parallel simulation to scale training across multiple CPUs, enabling fast prototyping and reproducibility.

Experimental results

Research questions

• RQ1 Can DRL effectively control the unstable falling liquid film despite its strong nonlinearity and high dimensionality?
• RQ2 How does incorporating translational invariance into the DRL agent architecture affect learning speed and policy performance?
• RQ3 To what extent does using a dense, local reward function improve training efficiency and control accuracy compared to sparse or global rewards?
• RQ4 Can the DRL agent control an arbitrary number of actuators without suffering from the curse of dimensionality in the output space?
• RQ5 How do architectural choices—such as convolutional structure and reward density—impact the scalability and robustness of DRL in physical systems?

Key findings

• The proposed DRL method with translational invariance and dense local rewards achieves significantly faster and more stable learning compared to baseline methods using global or sparse rewards.
• The convolutional DRL agent successfully controls an arbitrary number of actuators, effectively overcoming the curse of dimensionality in the control output space.
• The dense reward function provides fine-grained feedback, enabling the agent to distinguish between effective and ineffective actions at individual actuator locations.
• Method M3 (dense reward) outperforms M1 and M2 in both learning speed and final policy quality, demonstrating the importance of localized feedback in high-dimensional control.
• The architecture's invariance to translation allows the agent to generalize across spatial positions, reducing sample complexity and improving generalization.
• The approach is scalable and extensible, with potential application to complex 2D and 3D fluid systems exhibiting similar physical invariants and locality.