[Paper Review] Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation

Ho Kei Cheng, Yu‐Wing Tai+1 more|arXiv (Cornell University)|Jun 9, 2021

Video Surveillance and Tracking Methods80 references132 citations

TL;DR

The paper introduces STCN, a memory-efficient space-time correspondence network that uses image-to-image affinity with negative squared Euclidean distance (L2) instead of dot product, enabling diversified memory voting and state-of-the-art, fast semi-supervised video object segmentation.

ABSTRACT

This paper presents a simple yet effective approach to modeling space-time correspondences in the context of video object segmentation. Unlike most existing approaches, we establish correspondences directly between frames without re-encoding the mask features for every object, leading to a highly efficient and robust framework. With the correspondences, every node in the current query frame is inferred by aggregating features from the past in an associative fashion. We cast the aggregation process as a voting problem and find that the existing inner-product affinity leads to poor use of memory with a small (fixed) subset of memory nodes dominating the votes, regardless of the query. In light of this phenomenon, we propose using the negative squared Euclidean distance instead to compute the affinities. We validated that every memory node now has a chance to contribute, and experimentally showed that such diversified voting is beneficial to both memory efficiency and inference accuracy. The synergy of correspondence networks and diversified voting works exceedingly well, achieves new state-of-the-art results on both DAVIS and YouTubeVOS datasets while running significantly faster at 20+ FPS for multiple objects without bells and whistles.

Motivation & Objective

• Motivate a simpler, more memory-efficient approach to space-time matching for semi-supervised VOS.
• Replace per-object memory readout in STM with a frame-to-frame affinity that is reused across objects.
• Investigate affinity functions and memory coverage to improve diversity and utilization of memory nodes.
• Demonstrate that L2-based affinity yields diversified voting and boosts both accuracy and speed.

Proposed method

• Construct a Space-Time Correspondence Network (STCN) with a Key Encoder (image input) and a Value Encoder (image and mask input).
• Compute frame-to-frame affinities using a single, mask-agnostic key affinity matrix learned from RGB relations.
• Use negative squared Euclidean distance (L2) as the similarity measure instead of dot product to diversify memory contributions.
• Aggregate memory readouts via a matrix multiplication with the affinity matrix to produce query features for decoding the segmentation mask.
• Employ memory management where memory keys are reused from queried frames and memory values are produced per object after mask generation.
• Maintain a lightweight decoder and skip connections to generate high-resolution masks, enabling multi-object soft aggregation.

Experimental results

Research questions

• RQ1 How to construct an efficient frame-to-frame affinity for VOS without object-specific memory banks?
• RQ2 Does replacing dot product with L2 similarity improve memory coverage and segmentation performance?
• RQ3 Can a simpler STCN framework achieve state-of-the-art results while maintaining higher inference speed?
• RQ4 What is the impact of memory management strategies on speed and accuracy in STCN?

Key findings

• STCN matches or exceeds state-of-the-art on DAVIS 2017 and YouTubeVOS while running at 20+ FPS for multiple objects.
• Using L2 similarity diversifies memory contributions, reducing memory usage inequality and increasing robustness.
• Frame-to-frame affinity with shared encoders enables faster inference because the value encoder is invoked fewer times than STM's memory encoder.
• Removing the last-frame temporary memory and relying on frame-wide affinities improves speed from around 12 FPS (STM) to over 16–20 FPS (STCN) in various configurations.
• STCN plus optional BL30K pretraining further improves YouTubeVOS and DAVIS scores compared to the baseline.