[Paper Review] Dual Prototype Attention for Unsupervised Video Object Segmentation

Suhwan Cho, Minhyeok Lee+5 more|arXiv (Cornell University)|Nov 22, 2022

Visual Attention and Saliency Detection4 citations

TL;DR

This paper proposes Dual Prototype Attention (DPA), a novel unsupervised video object segmentation framework that introduces inter-modality attention (IMA) for adaptive fusion of appearance and motion features, and inter-frame attention (IFA) for global video context propagation via a prototype-based memory bank. The method achieves state-of-the-art performance on DAVIS 2016, FBMS, and YouTube-Objects benchmarks with superior accuracy and efficiency, outperforming prior methods by a significant margin while maintaining low computational cost.

ABSTRACT

Unsupervised video object segmentation (VOS) aims to detect and segment the most salient object in videos. The primary techniques used in unsupervised VOS are 1) the collaboration of appearance and motion information; and 2) temporal fusion between different frames. This paper proposes two novel prototype-based attention mechanisms, inter-modality attention (IMA) and inter-frame attention (IFA), to incorporate these techniques via dense propagation across different modalities and frames. IMA densely integrates context information from different modalities based on a mutual refinement. IFA injects global context of a video to the query frame, enabling a full utilization of useful properties from multiple frames. Experimental results on public benchmark datasets demonstrate that our proposed approach outperforms all existing methods by a substantial margin. The proposed two components are also thoroughly validated via ablative study.

Motivation & Objective

• To address the limitations of existing unsupervised video object segmentation methods in multi-modality fusion and temporal aggregation.
• To improve robustness against unreliable appearance or motion cues by enabling dense, mutual refinement between modalities.
• To efficiently leverage global video context without high computational cost, avoiding reliance on only the first frame or iterative refinement.
• To enhance feature representation by converting pixel-level features into structured prototypes with spatial knowledge.
• To achieve state-of-the-art performance on standard unsupervised VOS benchmarks while maintaining inference efficiency.

Proposed method

• Introduces inter-modality attention (IMA), which performs mutual feature refinement between appearance and motion branches using adaptive attention-based feature propagation.
• Employs inter-frame attention (IFA), which stores features from sampled frames in an external memory bank and adaptively retrieves global context for each query frame.
• Uses a prototype-based framework to convert pixel-level features into prototype-level representations, enhancing spatial structure awareness and robustness.
• Adopts a two-stream encoder-decoder architecture processing RGB images and optical flow maps in parallel, with IMA and IFA modules integrated at key stages.
• Utilizes a memory bank mechanism to store and retrieve features from multiple frames, enabling full video context utilization without iterative refinement.
• Applies adaptive attention mechanisms that dynamically allocate feature importance based on cross-modality and cross-frame relevance.

Experimental results

Research questions

• RQ1 Can a dual-attention mechanism improve multi-modality fusion in unsupervised video object segmentation by enabling dense, mutual refinement between appearance and motion features?
• RQ2 Can inter-frame attention effectively inject global video context into each query frame without relying on the initial frame or iterative refinement, thus improving temporal consistency?
• RQ3 Does incorporating a prototype-based representation learning framework enhance feature quality and robustness in unsupervised VOS?
• RQ4 Can the proposed method achieve state-of-the-art performance on standard benchmarks while maintaining low computational complexity?
• RQ5 How do the individual components (IMA, IFA, prototype framework) contribute to the overall performance gain in ablation studies?

Key findings

• On the DAVIS 2016 validation set, DPA achieves a mean JIoU score of 87.6% at 512×512 resolution, surpassing the previous SOTA method HFAN by 0.3%.
• On the FBMS test set, DPA achieves a 1.1% higher JIoU score than the next best method, demonstrating strong robustness in multi-object scenarios.
• On the challenging YouTube-Objects dataset, DPA ranks first in 4 out of 10 classes and achieves a mean JIoU of 87.4%, outperforming all prior methods.
• Ablation studies confirm that both IMA and IFA contribute significantly to performance, with the prototype framework further enhancing feature quality and generalization.
• The model maintains high inference efficiency, with a runtime of 25.4 seconds per video at 352×352 resolution on a single RTX 3090 GPU, outperforming methods with higher computational costs.
• Qualitative results show DPA maintains stable tracking under rapid motion and occlusion, while other methods fail due to background distractions.