[论文解读] Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation

Ho Kei Cheng, Yu‐Wing Tai+1 more|arXiv (Cornell University)|Jun 9, 2021

Video Surveillance and Tracking Methods参考文献 80被引用 132

一句话总结

论文提出 STCN，一种内存高效的时空对应网络，使用 image-to-image affinity 与负平方欧几里得距离（L2）而非点积，能够实现多样化的内存投票和最先进且快速的半监督视频对象分割。

ABSTRACT

This paper presents a simple yet effective approach to modeling space-time correspondences in the context of video object segmentation. Unlike most existing approaches, we establish correspondences directly between frames without re-encoding the mask features for every object, leading to a highly efficient and robust framework. With the correspondences, every node in the current query frame is inferred by aggregating features from the past in an associative fashion. We cast the aggregation process as a voting problem and find that the existing inner-product affinity leads to poor use of memory with a small (fixed) subset of memory nodes dominating the votes, regardless of the query. In light of this phenomenon, we propose using the negative squared Euclidean distance instead to compute the affinities. We validated that every memory node now has a chance to contribute, and experimentally showed that such diversified voting is beneficial to both memory efficiency and inference accuracy. The synergy of correspondence networks and diversified voting works exceedingly well, achieves new state-of-the-art results on both DAVIS and YouTubeVOS datasets while running significantly faster at 20+ FPS for multiple objects without bells and whistles.

研究动机与目标

• Motivate a simpler, more memory-efficient approach to space-time matching for semi-supervised VOS.
• Replace per-object memory readout in STM with a frame-to-frame affinity that is reused across objects.
• Investigate affinity functions and memory coverage to improve diversity and utilization of memory nodes.
• Demonstrate that L2-based affinity yields diversified voting and boosts both accuracy and speed.

提出的方法

• Construct a Space-Time Correspondence Network (STCN) with a Key Encoder (image input) and a Value Encoder (image and mask input).
• Compute frame-to-frame affinities using a single, mask-agnostic key affinity matrix learned from RGB relations.
• Use negative squared Euclidean distance (L2) as the similarity measure instead of dot product to diversify memory contributions.
• Aggregate memory readouts via a matrix multiplication with the affinity matrix to produce query features for decoding the segmentation mask.
• Employ memory management where memory keys are reused from queried frames and memory values are produced per object after mask generation.
• Maintain a lightweight decoder and skip connections to generate high-resolution masks, enabling multi-object soft aggregation.

实验结果

研究问题

• RQ1 How to construct an efficient frame-to-frame affinity for VOS without object-specific memory banks?
• RQ2 Does replacing dot product with L2 similarity improve memory coverage and segmentation performance?
• RQ3 Can a simpler STCN framework achieve state-of-the-art results while maintaining higher inference speed?
• RQ4 What is the impact of memory management strategies on speed and accuracy in STCN?

主要发现

Method	G (YouTubeVOS)	J_S	F_S	J_U	F_U	J&F	J	F	FPS
Ours	83.0	81.9	86.5	77.9	85.7	85.4	82.2	88.6	20.2
Ours + BL30K	84.3	83.2	87.9	79.0	87.3	85.3	82.0	88.6	20.2

• STCN matches or exceeds state-of-the-art on DAVIS 2017 and YouTubeVOS while running at 20+ FPS for multiple objects.
• Using L2 similarity diversifies memory contributions, reducing memory usage inequality and increasing robustness.
• Frame-to-frame affinity with shared encoders enables faster inference because the value encoder is invoked fewer times than STM's memory encoder.
• Removing the last-frame temporary memory and relying on frame-wide affinities improves speed from around 12 FPS (STM) to over 16–20 FPS (STCN) in various configurations.
• STCN plus optional BL30K pretraining further improves YouTubeVOS and DAVIS scores compared to the baseline.