[Paper Review] Aeolus Ocean -- A simulation environment for the autonomous COLREG-compliant navigation of Unmanned Surface Vehicles using Deep Reinforcement Learning and Maritime Object Detection

Andrew Alexander Vekinis, Stavros Perantonis|arXiv (Cornell University)|Jul 13, 2023

Maritime Navigation and SafetyEngineering3 citations

TL;DR

This paper introduces Aeolus Ocean, a high-fidelity simulation environment for training Deep Reinforcement Learning (DRL)-based autonomous navigation in Unmanned Surface Vehicles (USVs) that comply with COLREGs. The system integrates maritime object detection for situational awareness and demonstrates successful waypoint navigation in open sea and complex encounter scenarios under realistic weather and lighting conditions, with a binary executable of the simulator and trained agents publicly available.

ABSTRACT

Heading towards navigational autonomy in unmanned surface vehicles (USVs) in the maritime sector can fundamentally lead towards safer waters as well as reduced operating costs, while also providing a range of exciting new capabilities for oceanic research, exploration and monitoring. However, achieving such a goal is challenging. USV control systems must, safely and reliably, be able to adhere to the international regulations for preventing collisions at sea (COLREGs) in encounters with other vessels as they navigate to a given waypoint while being affected by realistic weather conditions, either during the day or at night. To deal with the multitude of possible scenarios, it is critical to have a virtual environment that is able to replicate the realistic operating conditions USVs will encounter, before they can be implemented in the real world. Such "digital twins" form the foundations upon which Deep Reinforcement Learning (DRL) and Computer Vision (CV) algorithms can be used to develop and guide USV control systems. In this paper we describe the novel development of a COLREG-compliant DRL-based collision avoidant navigational system with CV-based awareness in a realistic ocean simulation environment. The performance of the trained autonomous Agents resulting from this approach is evaluated in several successful navigations to set waypoints in both open sea and coastal encounters with other vessels. A binary executable version of the simulator with trained agents is available at https://github.com/aavek/Aeolus-Ocean

Motivation & Objective

• To develop a realistic, high-fidelity simulation environment for training autonomous USVs in complex maritime scenarios.
• To enable COLREG-compliant navigation through Deep Reinforcement Learning (DRL) with real-time situational awareness via maritime object detection.
• To validate the performance of trained DRL agents in diverse navigation tasks, including open-sea and coastal encounters with other vessels.
• To provide a publicly accessible binary version of the simulator and trained agents for reproducibility and further research.
• To bridge the gap between simulation and real-world deployment by modeling realistic weather, lighting, and dynamic vessel interactions.

Proposed method

• The simulation environment, Aeolus Ocean, replicates realistic oceanic conditions including wind, waves, and daylight/darkness cycles.
• A DRL agent is trained to navigate USVs to a target waypoint while avoiding collisions, using a reward function that prioritizes COLREG compliance.
• Maritime object detection is integrated via a computer vision pipeline to detect and track other vessels, enabling real-time situational awareness.
• The DRL agent processes visual observations from simulated cameras to make navigation decisions, ensuring compliance with COLREGs rules.
• The environment supports both open-sea and coastal navigation scenarios with dynamic, non-player-controlled vessels following realistic traffic patterns.
• A binary executable of the simulator with trained agents is released, enabling immediate deployment and benchmarking.

Experimental results

Research questions

• RQ1 Can a DRL-based navigation system achieve COLREG-compliant behavior in complex, dynamic maritime environments?
• RQ2 How effective is the integration of maritime object detection in enabling real-time situational awareness for autonomous USVs?
• RQ3 Can the DRL agent generalize across diverse scenarios, including open-sea navigation and close-proximity encounters with other vessels?
• RQ4 How does the simulator’s realism impact the generalization and robustness of the trained agents?
• RQ5 To what extent can the trained agents achieve safe and efficient waypoint navigation under realistic weather and lighting conditions?

Key findings

• The DRL agent successfully navigated to waypoints in both open-sea and coastal scenarios without violating COLREGs rules.
• The integration of maritime object detection significantly improved the agent’s ability to detect and respond to nearby vessels in real time.
• The agent demonstrated robust performance across various weather conditions, including daylight and nighttime operations.
• The simulator environment enabled stable and reproducible training, with the trained agents generalizing well to unseen scenarios.
• The release of a binary executable with trained agents facilitates immediate use and benchmarking in real-world testing and further research.