[Paper Review] Large Language Model-based Decision-making for COLREGs and the Control of Autonomous Surface Vehicles

Klinsmann Agyei, Pouria Sarhadi+1 more|arXiv (Cornell University)|Nov 25, 2024

Risk and Safety AnalysisDecision Sciences3 citations

TL;DR

This paper presents the first LLM-based decision-making framework for autonomous surface vehicles (ASVs) that ensures real-time compliance with COLREGs rules through explainable, risk-aware reasoning. By integrating online collision risk assessment with LLM-generated, human-interpretable decisions, the system achieves COLREGs-compliant maneuvers—such as proper give-way or stand-on actions—while maintaining accurate waypoint tracking in dynamic maritime scenarios.

ABSTRACT

In the field of autonomous surface vehicles (ASVs), devising decision-making and obstacle avoidance solutions that address maritime COLREGs (Collision Regulations), primarily defined for human operators, has long been a pressing challenge. Recent advancements in explainable Artificial Intelligence (AI) and machine learning have shown promise in enabling human-like decision-making. Notably, significant developments have occurred in the application of Large Language Models (LLMs) to the decision-making of complex systems, such as self-driving cars. The textual and somewhat ambiguous nature of COLREGs (from an algorithmic perspective), however, poses challenges that align well with the capabilities of LLMs, suggesting that LLMs may become increasingly suitable for this application soon. This paper presents and demonstrates the first application of LLM-based decision-making and control for ASVs. The proposed method establishes a high-level decision-maker that uses online collision risk indices and key measurements to make decisions for safe manoeuvres. A tailored design and runtime structure is developed to support training and real-time action generation on a realistic ASV model. Local planning and control algorithms are integrated to execute the commands for waypoint following and collision avoidance at a lower level. To the authors' knowledge, this study represents the first attempt to apply explainable AI to the dynamic control problem of maritime systems recognising the COLREGs rules, opening new avenues for research in this challenging area. Results obtained across multiple test scenarios demonstrate the system's ability to maintain online COLREGs compliance, accurate waypoint tracking, and feasible control, while providing human-interpretable reasoning for each decision.

Motivation & Objective

• Address the challenge of enabling autonomous surface vehicles (ASVs) to comply with the International Regulations for Preventing Collisions at Sea (COLREGs), which are written in natural language and require human-like reasoning.
• Overcome limitations of traditional rule-based systems by introducing explainable AI via Large Language Models (LLMs) to interpret and apply COLREGs in real time.
• Develop a hybrid architecture combining high-level LLM-based decision-making with low-level local planning and control for safe, compliant, and traceable ASV navigation.
• Ensure transparency and trust in autonomous maritime systems by generating human-readable justifications for each maneuver decision.
• Demonstrate the feasibility and effectiveness of LLMs in dynamic, complex maritime encounter scenarios, including overtaking, crossing, and head-on situations.

Proposed method

• Employ a high-level LLM-based decision maker that processes real-time inputs including range, bearing, CPA (Closest Point of Approach), T_CPA (Time to CPA), and online collision risk indices to assess encounter situations.
• Design a prompt engineering framework that structures LLM inputs with contextual parameters and rules, enabling accurate interpretation of COLREGs rules such as Rule 15 (giving way) and Rule 17 (stand-on vessel duties).
• Integrate the LLM’s action output with a low-level control system that executes waypoint following and collision avoidance using local planning algorithms.
• Use a nonlinear ASV model with realistic dynamics, including input saturation, disturbances, and non-holonomic constraints, to simulate real-world maritime behavior.
• Implement a runtime architecture that supports both training and real-time inference, ensuring responsiveness and consistency in decision-making.
• Apply explainable AI principles by generating natural language justifications for each maneuver, explicitly referencing relevant COLREGs rules and critical parameters like risk value, D_CPA, and T_CPA.

Experimental results

Research questions

• RQ1 Can Large Language Models (LLMs) accurately interpret and apply the natural language-based COLREGs rules in real-time maritime decision-making for ASVs?
• RQ2 How can LLM-based decision-making be integrated with low-level control systems to ensure both safety and compliance with COLREGs in dynamic encounter scenarios?
• RQ3 To what extent can LLMs provide human-interpretable, traceable reasoning for autonomous navigation decisions, enhancing transparency and trust?
• RQ4 Can risk-aware LLM decision-making prevent oscillatory or inconsistent maneuvers while maintaining consistent COLREGs compliance across diverse encounter types?
• RQ5 How does the integration of online collision risk assessment with LLM reasoning improve situational awareness and decision quality in complex maritime environments?

Key findings

• The LLM-based system successfully classified encounter scenarios (e.g., overtaking, crossing) and applied the correct COLREGs rule—such as Rule 15 for give-way or Rule 17 for stand-on—based on relative bearing and risk parameters.
• In a crossing situation with the target on the starboard side (relative heading -150.09°), the system correctly identified the own vessel as the give-way vessel and decided to 'Give-way, turn starboard' due to high risk, low D_CPA (12.4m), and critical T_CPA (3.2s).
• In a crossing stand-on scenario with the target on the port side (relative heading 125.08°), the system maintained stand-on behavior with a moderate risk value (0.78), T_CPA of 53.27s, and D_CPA of 296.21m, correctly justifying this with 'Risk present but manageable, maintaining stand-on duty as target vessel on port side.'
• The system demonstrated consistent, non-oscillatory behavior across all test scenarios, avoiding erratic maneuvers by relying on risk-aware decisioning and rule-based consistency.
• The integration of online risk indices (e.g., CPA-based risk) with LLM reasoning enabled the system to balance safety, compliance, and mission objectives, resulting in feasible and predictable control actions.
• The framework produced fully explainable decision logs, with each maneuver accompanied by a natural language rationale referencing specific COLREGs rules and key parameters, enhancing transparency and auditability.