[Paper Review] RiWNet: A moving object instance segmentation Network being Robust in adverse Weather conditions

Chenjie Wang, Chengyuan Li+3 more|arXiv (Cornell University)|Sep 4, 2021

Advanced Neural Network Applications61 references4 citations

TL;DR

RiWNet is a novel end-to-end moving object instance segmentation network designed to be robust in adverse weather conditions by integrating a new Feature Pyramid Network (RiWFPN) that enhances low-frequency structural features through progressive top-down interaction and attention refinement. The method improves performance in rain, fog, and low-light scenarios without requiring weather-specific data augmentation or multi-sensor fusion, achieving state-of-the-art results on benchmark datasets under challenging weather conditions.

ABSTRACT

Segmenting each moving object instance in a scene is essential for many applications. But like many other computer vision tasks, this task performs well in optimal weather, but then adverse weather tends to fail. To be robust in weather conditions, the usual way is to train network in data of given weather pattern or to fuse multiple sensors. We focus on a new possibility, that is, to improve its resilience to weather interference through the network's structural design. First, we propose a novel FPN structure called RiWFPN with a progressive top-down interaction and attention refinement module. RiWFPN can directly replace other FPN structures to improve the robustness of the network in non-optimal weather conditions. Then we extend SOLOV2 to capture temporal information in video to learn motion information, and propose a moving object instance segmentation network with RiWFPN called RiWNet. Finally, in order to verify the effect of moving instance segmentation in different weather disturbances, we propose a VKTTI-moving dataset which is a moving instance segmentation dataset based on the VKTTI dataset, taking into account different weather scenes such as rain, fog, sunset, morning as well as overcast. The experiment proves how RiWFPN improves the network's resilience to adverse weather effects compared to other FPN structures. We compare RiWNet to several other state-of-the-art methods in some challenging datasets, and RiWNet shows better performance especially under adverse weather conditions.

Motivation & Objective

• To address the degradation of moving object instance segmentation under adverse weather conditions such as rain, fog, and low light.
• To improve network robustness without relying on weather-augmented training data or multi-sensor fusion.
• To develop a feature pyramid network (RiWFPN) that enhances low-frequency structural information for better object representation in degraded images.
• To extend SOLOv2 with temporal modeling to capture motion cues from adjacent RGB frames for improved instance segmentation.
• To create and evaluate on a new benchmark dataset, VKTTI-moving, which includes diverse weather conditions for moving object instance segmentation.

Proposed method

• Proposes RiWFPN, a novel FPN variant with a progressive top-down interaction module that refines feature maps across scales to improve semantic and spatial consistency.
• Introduces an attention refinement module that enhances low-frequency components in feature maps to preserve structural information under weather degradation.
• Extends SOLOv2 with a ConvLSTM-based temporal encoder to incorporate motion cues from a pair of adjacent RGB frames.
• Uses a pair of consecutive RGB frames as input to RiWNet, enabling the network to leverage spatiotemporal context for more accurate instance segmentation.
• Constructs the VKTTI-moving dataset by re-labeling the VKITTI dataset with instance segmentation annotations across diverse weather conditions including rain, fog, sunset, morning, and overcast.
• Trains and evaluates RiWNet end-to-end on VKTTI-moving and YTVOS-moving, using standard metrics including precision, recall, F-measure, and IoU.

Experimental results

Research questions

• RQ1 Can a redesigned FPN architecture improve the robustness of instance segmentation networks under adverse weather conditions without requiring weather-augmented data?
• RQ2 How effective is the integration of progressive top-down feature refinement and attention-based low-frequency enhancement in preserving structural information in degraded images?
• RQ3 To what extent can temporal modeling via ConvLSTM improve instance segmentation performance in video sequences under weather disturbances?
• RQ4 How does RiWNet compare to state-of-the-art methods in moving object instance segmentation under challenging weather conditions?
• RQ5 Can a new benchmark dataset with diverse weather conditions effectively evaluate and validate the robustness of segmentation models?

Key findings

• RiWNet achieves a precision of 79.80% and an F-measure of 73.35% on the YTVOS-moving test set, outperforming TSA and U2-ONet by over 4.1% in precision and 3.4% in F-measure.
• The use of RiWFPN significantly improves feature map quality under adverse weather, as evidenced by cleaner, more structured features even in rainy and foggy conditions.
• In visual-LiDAR fusion SLAM (DV-LOAM), integrating RiWNet reduces relative translational error from 0.61% to 0.56% and rotational error from 0.30°/100m to 0.26°/100m on KITTI Sequence 04.
• The point cloud maps generated with RiWNet’s moving object masks show a substantial reduction in moving object artifacts, improving static scene reconstruction quality.
• RiWNet maintains real-time inference at ~5 Hz when running in parallel on keyframes, demonstrating compatibility with real-time SLAM systems.
• The VKTTI-moving dataset effectively captures the performance gap between standard and robust segmentation models under diverse weather conditions, validating its utility as a benchmark.