[Paper Review] Symmetry reduction for deep reinforcement learning active control of chaotic spatiotemporal dynamics

Kevin Zeng, Michael D. Graham|arXiv (Cornell University)|Apr 9, 2021

Model Reduction and Neural Networks39 references33 citations

TL;DR

This paper proposes symmetry-reduced deep reinforcement learning (RL) to improve data efficiency and control efficacy in chaotic spatiotemporal systems, using the Kuramoto-Sivashinsky equation as a testbed. By projecting the state-action space into a symmetry-invariant manifold, the method enables faster learning, stabilizes an equilibrium state connected to the unforced system, and achieves robustness to noise and parameter variations.

ABSTRACT

Deep reinforcement learning (RL) is a data-driven, model-free method capable of discovering complex control strategies for macroscopic objectives in high-dimensional systems, making its application towards flow control promising. Many systems of flow control interest possess symmetries that, when neglected, can significantly inhibit the learning and performance of a naive deep RL approach. Using a test-bed consisting of the Kuramoto-Sivashinsky Equation (KSE), equally spaced actuators, and a goal of minimizing dissipation and power cost, we demonstrate that by moving the deep RL problem to a symmetry-reduced space, we can alleviate limitations inherent in the naive application of deep RL. We demonstrate that symmetry-reduced deep RL yields improved data efficiency as well as improved control policy efficacy compared to policies found by naive deep RL. Interestingly, the policy learned by the the symmetry aware control agent drives the system toward an equilibrium state of the forced KSE that is connected by continuation to an equilibrium of the unforced KSE, despite having been given no explicit information regarding its existence. I.e., to achieve its goal, the RL algorithm discovers and stabilizes an equilibrium state of the system. Finally, we demonstrate that the symmetry-reduced control policy is robust to observation and actuation signal noise, as well as to system parameters it has not observed before.

Motivation & Objective

• To address the poor data efficiency and performance limitations of naive deep RL in high-dimensional, symmetric chaotic systems.
• To investigate whether symmetry-aware RL can discover and stabilize equilibria in chaotic spatiotemporal dynamics.
• To improve control policy efficacy by reducing the state space using continuous and discrete symmetries of the system.
• To evaluate robustness of the symmetry-reduced policy under noise and unseen system parameters.
• To demonstrate that symmetry reduction enables discovery of non-trivial control strategies without prior knowledge of equilibrium states.

Proposed method

• The method projects the system's state and actions into a symmetry-reduced space using the continuous translational and discrete reflection symmetries of the Kuramoto-Sivashinsky equation (KSE).
• A deep Q-network (DQN) agent is trained in the symmetry-reduced state-action space to minimize time-averaged dissipation and actuation cost.
• The symmetry reduction is implemented via a coordinate transformation that eliminates redundant dynamical states related by symmetry.
• The policy is trained using a reward function that penalizes high dissipation and control energy, encouraging low-dissipation states.
• The approach avoids explicit symmetry constraints in the network architecture, instead relying on the reduced state space to implicitly enforce invariance.
• Robustness is tested under observation noise, actuation noise, and parameter variations not seen during training.

Experimental results

Research questions

• RQ1 Can symmetry reduction in deep RL improve data efficiency and control performance in chaotic spatiotemporal systems?
• RQ2 Does symmetry-aware RL discover and stabilize an equilibrium state of the forced KSE that is connected to the unforced system's equilibrium?
• RQ3 How does the symmetry-reduced policy compare to naive deep RL in terms of convergence speed and final performance?
• RQ4 Is the symmetry-reduced policy robust to noise in observations and actuation signals?
• RQ5 Can the policy generalize to system parameters not encountered during training?

Key findings

• Symmetry-reduced deep RL achieves faster convergence and improved data efficiency compared to naive deep RL in controlling the KSE.
• The symmetry-aware agent stabilizes a state connected to the unforced KSE's equilibrium, despite no explicit information about this state being provided.
• The learned policy reduces time-averaged dissipation by over 50% compared to uncontrolled dynamics and outperforms naive RL policies.
• The symmetry-reduced policy remains effective under 10% observation noise and 10% actuation noise, demonstrating robustness.
• The policy generalizes to system parameters (e.g., forcing amplitude) outside the training range, indicating strong generalization capability.
• The method enables discovery of a non-trivial control strategy that stabilizes a system equilibrium, suggesting utility as a discovery tool in complex fluid dynamics.