[Paper Review] Video Object Segmentation with Adaptive Feature Bank and Uncertain-Region Refinement

Yongqing Liang, Xin Li+2 more|arXiv (Cornell University)|Oct 15, 2020

Visual Attention and Saliency DetectionComputer Science33 references78 citations

TL;DR

Proposes an adaptive feature bank and uncertain-region refinement to enhance semi-supervised video object segmentation, achieving state-of-the-art results on DAVIS16/17 and YouTube-VOS benchmarks without online fine-tuning.

ABSTRACT

We propose a new matching-based framework for semi-supervised video object segmentation (VOS). Recently, state-of-the-art VOS performance has been achieved by matching-based algorithms, in which feature banks are created to store features for region matching and classification. However, how to effectively organize information in the continuously growing feature bank remains under-explored, and this leads to inefficient design of the bank. We introduce an adaptive feature bank update scheme to dynamically absorb new features and discard obsolete features. We also design a new confidence loss and a fine-grained segmentation module to enhance the segmentation accuracy in uncertain regions. On public benchmarks, our algorithm outperforms existing state-of-the-arts.

Motivation & Objective

• Motivate improved memory management for matching-based VOS in long videos.
• Develop an adaptive feature bank that absorbs new features and discards obsolete ones.
• Introduce an uncertain-region refinement module to better handle boundary ambiguities.
• Train a framework combining cross-entropy loss with a confidence-based loss to reduce segmentation ambiguity.
• Demonstrate robustness and efficiency on long videos and multiple benchmarks.

Proposed method

• Use a matching-based segmentation pipeline with a query encoder and per-object feature banks.
• Maintain an adaptive feature bank for each object that merges similar features and appends distinct ones (Equations 3–5).
• Discard obsolete features via an LFU-based policy to keep memory within budget (Equation 6).
• Introduce an uncertain-region refinement (URR) that computes a pixel-wise uncertainty map and a confidence loss (Equations 7–9).
• Apply a local refinement mechanism using neighborhood references to refine ambiguous boundaries (Equations 10–12).
• Train with a combination of classification loss and confidence loss without online fine-tuning.

Experimental results

Research questions

• RQ1 How can an adaptive feature bank efficiently manage long-video object memories for matching-based VOS?
• RQ2 Does incorporating a confidence-driven uncertain-region refinement improve boundary accuracy and overall segmentation quality?
• RQ3 Can the proposed method surpass state-of-the-art on standard VOS benchmarks without online fine-tuning?

Key findings

• Outperforms state-of-the-art methods on DAVIS17 and YouTube-VOS benchmarks without online fine-tuning.
• Demonstrates strong performance for long videos with dynamic feature banking and LFU-based pruning.
• URR improves boundary and uncertain region segmentation through a confidence loss and local refinement.
• AFB provides memory efficiency, enabling handling of long videos with competitive runtime.
• Ablation shows the full framework (AFB+URR) yields the best J&F scores compared to variants.
• On DAVIS17, the method achieves J&F of 74.6 without online fine-tuning and 4.0 fps on 1080Ti.