[Paper Review] Mask Propagation for Efficient Video Semantic Segmentation

Yuetian Weng, Mingfei Han+5 more|arXiv (Cornell University)|Oct 29, 2023

Advanced Image and Video Retrieval Techniques4 citations

TL;DR

This paper proposes MPVSS, a novel mask propagation framework for efficient video semantic segmentation that leverages query-based image segmentation on sparse key frames and learns segment-aware flow maps for accurate, efficient mask warping to non-key frames. The method achieves state-of-the-art accuracy and efficiency, outperforming SOTA models like MRCFA by 4.0% mIoU with only 26% of the FLOPs on VSPW, while reducing FLOPs by up to 4× compared to per-frame baselines with minimal accuracy drop.

ABSTRACT

Video Semantic Segmentation (VSS) involves assigning a semantic label to each pixel in a video sequence. Prior work in this field has demonstrated promising results by extending image semantic segmentation models to exploit temporal relationships across video frames; however, these approaches often incur significant computational costs. In this paper, we propose an efficient mask propagation framework for VSS, called MPVSS. Our approach first employs a strong query-based image segmentor on sparse key frames to generate accurate binary masks and class predictions. We then design a flow estimation module utilizing the learned queries to generate a set of segment-aware flow maps, each associated with a mask prediction from the key frame. Finally, the mask-flow pairs are warped to serve as the mask predictions for the non-key frames. By reusing predictions from key frames, we circumvent the need to process a large volume of video frames individually with resource-intensive segmentors, alleviating temporal redundancy and significantly reducing computational costs. Extensive experiments on VSPW and Cityscapes demonstrate that our mask propagation framework achieves SOTA accuracy and efficiency trade-offs. For instance, our best model with Swin-L backbone outperforms the SOTA MRCFA using MiT-B5 by 4.0% mIoU, requiring only 26% FLOPs on the VSPW dataset. Moreover, our framework reduces up to 4x FLOPs compared to the per-frame Mask2Former baseline with only up to 2% mIoU degradation on the Cityscapes validation set. Code is available at https://github.com/ziplab/MPVSS.

Motivation & Objective

• To address the high computational cost of processing every video frame with strong image segmentors in video semantic segmentation (VSS).
• To reduce temporal redundancy in video processing by reusing accurate mask predictions from sparse key frames.
• To improve flow estimation for mask warping by incorporating segment-level information rather than relying on pixel-level optical flow.
• To achieve a superior accuracy-efficiency trade-off in VSS without sacrificing category consistency across frames.

Proposed method

• A strong query-based image segmentor (e.g., Mask2Former) is applied only to sparse key frames to generate accurate binary masks and class predictions.
• A query-based flow estimation module is proposed, where each query from the key frame generates a segment-aware flow map specific to its corresponding mask.
• The flow maps are learned using the same query embeddings from the key frame, enabling motion feature extraction per segment and improving flow accuracy.
• Mask predictions from key frames are warped to non-key frames using the predicted query-based flow maps, replacing per-frame inference.
• A refinement step enhances pixel-wise flow using segment-level motion cues, further improving warping quality.
• The framework supports flexible key frame intervals, enabling dynamic trade-offs between accuracy and computational cost.

Figure 1 : Motivation of the proposed MPVSS. Three consecutive frames with ground truth category labels are sampled from a video in VSPW [ 42 ] , illustrating a strong correlation between video frames along the temporal dimension. This observation underscores the significant temporal redundancy pres

Experimental results

Research questions

• RQ1 Can segment-aware flow estimation improve mask propagation accuracy in video semantic segmentation compared to traditional optical flow?
• RQ2 How does reusing mask predictions from key frames affect the efficiency and accuracy of video semantic segmentation?
• RQ3 To what extent can query-based flow learning reduce computational cost while maintaining high mIoU performance?
• RQ4 How robust is the proposed method to increasing key frame intervals in terms of accuracy degradation?

Key findings

• On the VSPW dataset, MPVSS with a Swin-L backbone achieves 4.0% higher mIoU than the SOTA MRCFA method while using only 26% of the FLOPs.
• Compared to the per-frame Mask2Former baseline, MPVSS reduces FLOPs by up to 4× with only a 2% mIoU degradation on the Cityscapes validation set.
• The proposed query-based flow consistently outperforms traditional optical flow, especially in long-range temporal propagation, with only 1.1% mIoU drop over a key frame interval of 10, compared to 2.63% for optical flow.
• Using learned queries for flow estimation improves mIoU by 1.1% and 1.5% on Swin-T and Swin-B, respectively, compared to random initialization.
• The Query-for-PF variant further improves performance by 0.1% and 0.2% over optical flow, demonstrating the benefit of segment-level information in flow learning.
• Qualitative results show that MPVSS maintains better category consistency and more accurately tracks small moving objects than per-frame baselines and optical flow-based methods.

Figure 2 : Overall architecture of the proposed MPVSS.