[Paper Review] Proportional integral derivative controller assisted reinforcement learning for path following by autonomous underwater vehicles

Simen Theie Havenstrøm, Camilla Sterud+2 more|arXiv (Cornell University)|Feb 3, 2020

Adaptive Dynamic Programming Control38 references4 citations

TL;DR

This paper proposes a novel deep reinforcement learning (DRL) approach for 3D path following in autonomous underwater vehicles (AUVs) by integrating proportional-integral-derivative (PID) controllers to assist DRL training. The PID-assisted method splits control into three actuators (rudder, elevator, thrust), enabling stable, efficient training with reduced time-steps and robust performance under unseen ocean currents, outperforming end-to-end DRL in stability and convergence.

ABSTRACT

Control theory provides engineers with a multitude of tools to design controllers that manipulate the closed-loop behavior and stability of dynamical systems. These methods rely heavily on insights about the mathematical model governing the physical system. However, if a system is highly complex, it might be infeasible to produce a reliable mathematical model of the system. Without a model most of the theoretical tools to develop control laws break down. In these settings, machine learning controllers become attractive: Controllers that can learn and adapt to complex systems, developing control laws where the engineer cannot. This article focuses on utilizing machine learning controllers in practical applications, specifically using deep reinforcement learning in motion control systems for an autonomous underwater vehicle with six degrees-of-freedom. Two methods are considered: end-to-end learning, where the vehicle is left entirely alone to explore the solution space in its search for an optimal policy, and PID assisted learning, where the DRL controller is essentially split into three separate parts, each controlling its own actuator.

Motivation & Objective

• Address the challenge of controlling 6-degree-of-freedom autonomous underwater vehicles (AUVs) in complex, uncertain underwater environments.
• Overcome limitations of traditional control methods that rely on accurate mathematical models and struggle with nonlinearities and disturbances.
• Explore data-driven machine learning control (MLC) as a viable alternative to model-based control for AUV path following.
• Improve training stability and convergence speed by integrating PID controllers as a supervisory layer in DRL training.
• Demonstrate robustness of the DRL controller to unseen disturbances such as varying ocean currents, validating generalization capability.

Proposed method

• Employ the Proximal Policy Optimization (PPO) algorithm for deep reinforcement learning (DRL) to train an agent to follow 3D spatial trajectories.
• Implement a PID-assisted learning framework where the DRL agent controls only one actuator at a time (rudder, elevator, or thrust), while the other two are stabilized by fixed PID controllers.
• Use a quadratic cost function as the reward signal to minimize tracking error and control effort.
• Train the DRL agent in a simulated environment with no ocean currents, then evaluate performance under unseen current disturbances.
• Compare end-to-end DRL (full agent control) with PID-assisted DRL (modular control per actuator) in terms of training efficiency, convergence, and robustness.
• Use synthetic data from a high-fidelity AUV simulation model to generate training trajectories and reward signals.

Experimental results

Research questions

• RQ1 Can DRL with PID-assisted control achieve faster and more stable training for 3D path following in AUVs compared to end-to-end DRL?
• RQ2 How does the DRL controller perform under unseen ocean current disturbances when trained without such conditions?
• RQ3 To what extent does the PID-assisted architecture reduce the number of simulation time-steps required for convergence?
• RQ4 Can the DRL agent generalize to novel, previously untrained environmental conditions without retraining?
• RQ5 What is the trade-off between interpretability and performance when using a black-box DNN versus a hybrid PID-DRL control structure?

Key findings

• The PID-assisted DRL approach significantly reduced the number of simulated time-steps required to train the controller compared to end-to-end learning.
• The DRL agent achieved stable 3D path following even when exposed to ocean currents of varying intensity, despite being trained in still water.
• A constant vertical control deviation was observed in the presence of currents, indicating partial compensation but not full disturbance rejection.
• The end-to-end DRL method showed promising results but suffered from high sensitivity to training setup and complex reward shaping.
• The PID-assisted method demonstrated robustness to unseen disturbances, suggesting the potential for generalization in real-world applications.
• The lack of a priori model knowledge and the self-tuning nature of DRL via exploration and exploitation were confirmed as key advantages over traditional control methods.