[Paper Review] Temporal Context for Robust Maritime Obstacle Detection

Lojze Žust, Matej Kristan|arXiv (Cornell University)|Mar 10, 2022

Maritime Navigation and Safety4 citations

TL;DR

This paper proposes WaSR-T, a temporal convolutional network that improves maritime obstacle detection by leveraging multi-frame temporal context to distinguish dynamic water reflections and sun glitter from real obstacles. By analyzing temporal appearance changes in recent frames, WaSR-T reduces false positives by 41% overall and 53% in the danger zone, achieving state-of-the-art performance on the MODS benchmark with minimal computational overhead.

ABSTRACT

Robust maritime obstacle detection is essential for fully autonomous unmanned surface vehicles (USVs). The currently widely adopted segmentation-based obstacle detection methods are prone to misclassification of object reflections and sun glitter as obstacles, producing many false positive detections, effectively rendering the methods impractical for USV navigation. However, water-turbulence-induced temporal appearance changes on object reflections are very distinctive from the appearance dynamics of true objects. We harness this property to design WaSR-T, a novel maritime obstacle detection network, that extracts the temporal context from a sequence of recent frames to reduce ambiguity. By learning the local temporal characteristics of object reflection on the water surface, WaSR-T substantially improves obstacle detection accuracy in the presence of reflections and glitter. Compared with existing single-frame methods, WaSR-T reduces the number of false positive detections by 41% overall and by over 53% within the danger zone of the boat, while preserving a high recall, and achieving new state-of-the-art performance on the challenging MODS maritime obstacle detection benchmark. The code, pretrained models and extended datasets are available at https://github.com/lojzezust/WaSR-T

Motivation & Objective

• To address the high rate of false positive detections in single-frame maritime obstacle detection caused by water reflections and sun glitter.
• To improve robustness in obstacle segmentation under challenging maritime conditions where appearance ambiguity is high.
• To develop a deep learning architecture that explicitly models temporal dynamics to differentiate transient reflections from static obstacles.
• To extend the MaSTr1325 dataset with temporally aligned frames and new reflection/glitter scenes to better train temporal models.
• To establish a new state-of-the-art in maritime obstacle detection using temporal context and a curated, extended training dataset.

Proposed method

• WaSR-T uses a shared encoder to extract feature maps from a target frame and preceding context frames.
• A 3D convolutional temporal context module processes the sequence of feature maps to learn spatiotemporal patterns of dynamic reflections.
• The network combines the temporal context embeddings with the target frame features before passing them to a decoder for segmentation prediction.
• The model is trained end-to-end on a modified version of the MaSTr1325 dataset, extended to MaSTr1478 with additional temporally aligned frames and challenging reflection scenes.
• The temporal context module uses a 3D convolution with variable kernel size and context length to capture local temporal dynamics of water surface deformations.
• The architecture is designed to preserve high recall while significantly reducing false positives, especially in the danger zone near the USV.

Experimental results

Research questions

• RQ1 Can modeling temporal context in a sequence of frames improve the robustness of maritime obstacle detection in the presence of reflections and glitter?
• RQ2 How does the inclusion of temporally extended training data affect the performance of a segmentation network in detecting obstacles on water?
• RQ3 What is the optimal temporal context length and spatial kernel size for distinguishing dynamic water reflections from real obstacles?
• RQ4 Does the addition of realistic reflection and glitter scenes to the training set improve generalization and reduce false positives?
• RQ5 To what extent does a temporal modeling approach outperform single-frame inference in real-world maritime obstacle detection?

Key findings

• WaSR-T reduces the number of false positive detections by 41% overall compared to single-frame WaSR, with a 53% reduction in the danger zone.
• The model achieves a 5.4-point increase in F1 score within the danger zone and a 1.0-point overall F1 improvement when trained on the extended MaSTr1478 dataset.
• Using just one context frame (T=1) reduces false positives by 30% overall and 39% in the danger zone, demonstrating the value of temporal context.
• A 3×3 spatial kernel size in the temporal context module yields optimal performance, while larger kernels provide no benefit.
• The extended dataset MaSTr1478 is essential for performance gains, as single-frame models do not benefit from the new sequences, and WaSR-T trained on MaSTr1325 alone shows no improvement.
• WaSR-T sets a new state-of-the-art on the MODS benchmark, outperforming existing methods that use or do not use temporal context.