[Paper Review] MODS -- A USV-oriented object detection and obstacle segmentation benchmark

Borja Bovcon, Jon Muhovič+4 more|arXiv (Cornell University)|May 5, 2021

Maritime Navigation and SafetyEngineering77 references92 citations

TL;DR

This paper introduces MODS, a new benchmark for unmanned surface vehicle (USV) perception that unifies maritime object detection and obstacle segmentation tasks. It presents a large-scale, IMU-synchronized stereo dataset of 80,828 images with 60,000+ annotated obstacles, along with a novel evaluation protocol that measures detection accuracy in a way meaningful for real-world USV navigation. The benchmark evaluates 19 state-of-the-art methods, revealing that general-purpose models underperform on small objects and that maritime-specific architectures like WaSR achieve superior false-positive control.

ABSTRACT

Small-sized unmanned surface vehicles (USV) are coastal water devices with a broad range of applications such as environmental control and surveillance. A crucial capability for autonomous operation is obstacle detection for timely reaction and collision avoidance, which has been recently explored in the context of camera-based visual scene interpretation. Owing to curated datasets, substantial advances in scene interpretation have been made in a related field of unmanned ground vehicles. However, the current maritime datasets do not adequately capture the complexity of real-world USV scenes and the evaluation protocols are not standardised, which makes cross-paper comparison of different methods difficult and hinders the progress. To address these issues, we introduce a new obstacle detection benchmark MODS, which considers two major perception tasks: maritime object detection and the more general maritime obstacle segmentation. We present a new diverse maritime evaluation dataset containing approximately 81k stereo images synchronized with an on-board IMU, with over 60k objects annotated. We propose a new obstacle segmentation performance evaluation protocol that reflects the detection accuracy in a way meaningful for practical USV navigation. Nineteen recent state-of-the-art object detection and obstacle segmentation methods are evaluated using the proposed protocol, creating a benchmark to facilitate development of the field. The proposed dataset, as well as evaluation routines, are made publicly available at vicos.si/resources.

Motivation & Objective

• To address the lack of standardized, large-scale, and diverse datasets for USV perception, particularly in obstacle detection and segmentation.
• To overcome the limitations of existing maritime datasets, which fail to capture real-world complexity and lack consistent evaluation protocols.
• To enable objective, cross-paper comparison of USV perception methods by introducing a unified benchmark with standardized evaluation metrics.
• To identify performance gaps in current state-of-the-art models when applied to real USV scenarios, especially for small and distant obstacles.
• To accelerate progress in autonomous USV navigation by providing publicly available data, training sets, and evaluation code.

Proposed method

• The authors collected a new stereo RGB dataset of 80,828 synchronized images from a USV operating under diverse weather conditions, with IMU time-stamping for precise synchronization.
• The dataset includes detailed annotations: bounding boxes for 60,000+ dynamic obstacles (e.g., swimmers, buoys, boats), and pixel-level segmentation masks for static obstacles (e.g., shorelines, piers) and navigable areas.
• A new obstacle segmentation evaluation protocol was designed to measure detection accuracy in a way meaningful for USV navigation, focusing on false-positive control and edge detection precision.
• The protocol computes key metrics such as true-positive rate (RTP), false-positive rate (RFP), and detection ratios (∇T, ∇F) within the danger zone near the USV.
• The benchmark includes curated training sets for both object detection and segmentation tasks, enabling consistent model training and evaluation.
• The evaluation is implemented in Python and applied to 19 state-of-the-art models, including Mask R-CNN, FCOS, and specialized segmentation networks like WaSR and BiSeNet.

Experimental results

Research questions

• RQ1 How do state-of-the-art object detection and segmentation models perform on a large-scale, real-world USV dataset with diverse environmental conditions?
• RQ2 What are the key failure modes of general-purpose models (e.g., COCO-trained) when applied to maritime obstacle detection, particularly for small or distant objects?
• RQ3 To what extent does geometric augmentation improve segmentation performance compared to color transfer in USV-specific training?
• RQ4 How does the proposed evaluation protocol reveal performance differences that are not captured by standard metrics like mAP?
• RQ5 Can segmentation-based obstacle detection methods achieve real-time performance and robust edge detection under real USV operation constraints?

Key findings

• General-purpose object detectors achieve approximately six times lower F1 scores on MODS compared to COCO, indicating significant domain-specific challenges.
• All methods suffer from low recall for small and very small objects, with the highest miss rates observed for swimmers and buoys, despite good overall precision.
• Mask R-CNN achieves the best overall detection with recognition performance, while FCOS performs best in detection-only tasks near the USV, achieving ~14 FPS on a high-end GPU.
• The maritime-specific method WaSR achieves the best F1 score in obstacle segmentation, primarily due to a very low false-positive rate (0.5% in danger zone) and high true-positive rate (98.8% for large objects).
• False positives in top methods are often tiny objects, suggesting that improving detection of small, distant obstacles is a key bottleneck.
• Segmentation models like RefineNet and BiSeNet detect water-obstacle boundaries robustly (>97% accuracy), but occasionally overshoot the edge, increasing collision risk.