[Paper Review] Multi-Source Fusion and Automatic Predictor Selection for Zero-Shot Video Object Segmentation

Xiaoqi Zhao, Youwei Pang+3 more|arXiv (Cornell University)|Aug 11, 2021

Visual Attention and Saliency Detection57 references4 citations

TL;DR

This paper proposes a multi-source fusion and automatic predictor selection framework for zero-shot video object segmentation, leveraging RGB, depth, optical flow, and static saliency features through an interoceptive spatial attention module (ISAM) and a feature purification module (FPM) to enhance feature fusion. An automatic predictor selection (APS) network dynamically chooses between static saliency and moving object predictions to mitigate errors from low-quality optical flow, achieving state-of-the-art performance on DAVIS 16, Youtube-Objects, and FBMS benchmarks.

ABSTRACT

Location and appearance are the key cues for video object segmentation. Many sources such as RGB, depth, optical flow and static saliency can provide useful information about the objects. However, existing approaches only utilize the RGB or RGB and optical flow. In this paper, we propose a novel multi-source fusion network for zero-shot video object segmentation. With the help of interoceptive spatial attention module (ISAM), spatial importance of each source is highlighted. Furthermore, we design a feature purification module (FPM) to filter the inter-source incompatible features. By the ISAM and FPM, the multi-source features are effectively fused. In addition, we put forward an automatic predictor selection network (APS) to select the better prediction of either the static saliency predictor or the moving object predictor in order to prevent over-reliance on the failed results caused by low-quality optical flow maps. Extensive experiments on three challenging public benchmarks (i.e. DAVIS$_{16}$, Youtube-Objects and FBMS) show that the proposed model achieves compelling performance against the state-of-the-arts. The source code will be publicly available at extcolor{red}{\url{https://github.com/Xiaoqi-Zhao-DLUT/Multi-Source-APS-ZVOS}}.

Motivation & Objective

• To address the limitation of existing zero-shot video object segmentation (ZVOS) methods that rely only on RGB or RGB and optical flow, by incorporating multiple complementary sources of information.
• To improve feature fusion by adaptively highlighting spatial importance of each source through an interoceptive spatial attention module (ISAM).
• To reduce inter-source interference by filtering incompatible features using a feature purification module (FPM).
• To mitigate performance degradation caused by low-quality optical flow by automatically selecting between static saliency and moving object predictions via a predictor selection network (APS).
• To achieve state-of-the-art performance on standard ZVOS benchmarks by integrating multi-source features and intelligent predictor selection.

Proposed method

• A multi-task network with a shared encoder and two decoders is used to predict depth and static saliency maps from RGB input, using the FPN architecture.
• The interoceptive spatial attention module (ISAM) adaptively computes attention weights for each source feature map based on spatial context, emphasizing more informative features.
• The feature purification module (FPM) decomposes the fused feature map into common and exclusive components, subtracting the exclusive part to suppress background noise and retain moving object cues.
• The automatic predictor selection (APS) network evaluates the reliability of optical flow-based predictions and selects between the static saliency predictor and moving object predictor based on confidence scores.
• The final prediction is generated by the selected predictor, ensuring robustness against low-quality optical flow.
• The framework is trained end-to-end on DAVIS 16, Youtube-Objects, and FBMS, with loss functions for depth, saliency, and segmentation supervision.

Experimental results

Research questions

• RQ1 How can multiple visual sources (RGB, depth, optical flow, static saliency) be effectively fused to improve zero-shot video object segmentation?
• RQ2 Can an attention mechanism that considers inter-source spatial relationships enhance feature representation in multi-source fusion?
• RQ3 How can incompatible or noisy features across sources be filtered to improve segmentation accuracy?
• RQ4 Can an automatic predictor selection mechanism reduce reliance on potentially unreliable optical flow and improve robustness?
• RQ5 Does the integration of multi-source features and intelligent predictor selection lead to state-of-the-art performance on standard ZVOS benchmarks?

Key findings

• The proposed method achieves a mean intersection-over-union (mIoU) of 83.4 and mean F-measure of 82.3 on the DAVIS 16 benchmark, outperforming previous state-of-the-art methods.
• On the Youtube-Objects dataset, the method achieves a mIoU of 74.9 and mean F-measure of 83.3, demonstrating strong generalization under low-quality optical flow conditions.
• The ISAM module improves mIoU by 2.1% and mean F-measure by 1.6% compared to simple concatenation of multi-source features.
• The FPM further boosts performance by 1.1% in mIoU and 1.1% in mean F-measure, confirming its effectiveness in filtering incompatible features.
• The APS network achieves superior performance by selecting the better predictor—SOS or MOS—based on optical flow quality, outperforming standalone SOS (72.7 mIoU) and MOS (70.2 mIoU) on Youtube-Objects.
• Visualizations confirm that APS assigns low scores to videos with low-quality optical flow and high scores to those with high-quality flow, validating its adaptive selection mechanism.