[Paper Review] Video K-Net: A Simple, Strong, and Unified Baseline for Video Segmentation

Xiangtai Li, Wenwei Zhang+5 more|arXiv (Cornell University)|Apr 10, 2022

Visual Attention and Saliency Detection4 citations

TL;DR

Video K-Net proposes a simple, unified, and end-to-end framework for video panoptic segmentation by leveraging learnable kernels from K-Net to jointly perform instance segmentation and tracking. It introduces kernel-based appearance modeling and cross-temporal kernel interaction with a contrastive loss for temporal association, achieving state-of-the-art performance on Cityscapes-VPS, KITTI-STEP, and VIPSeg with 12% relative improvement on KITTI-STEP and 15% on VIPSeg.

ABSTRACT

This paper presents Video K-Net, a simple, strong, and unified framework for fully end-to-end video panoptic segmentation. The method is built upon K-Net, a method that unifies image segmentation via a group of learnable kernels. We observe that these learnable kernels from K-Net, which encode object appearances and contexts, can naturally associate identical instances across video frames. Motivated by this observation, Video K-Net learns to simultaneously segment and track "things" and "stuff" in a video with simple kernel-based appearance modeling and cross-temporal kernel interaction. Despite the simplicity, it achieves state-of-the-art video panoptic segmentation results on Citscapes-VPS, KITTI-STEP, and VIPSeg without bells and whistles. In particular, on KITTI-STEP, the simple method can boost almost 12\% relative improvements over previous methods. On VIPSeg, Video K-Net boosts almost 15\% relative improvements and results in 39.8 % VPQ. We also validate its generalization on video semantic segmentation, where we boost various baselines by 2\% on the VSPW dataset. Moreover, we extend K-Net into clip-level video framework for video instance segmentation, where we obtain 40.5% mAP for ResNet50 backbone and 54.1% mAP for Swin-base on YouTube-2019 validation set. We hope this simple, yet effective method can serve as a new, flexible baseline in unified video segmentation design. Both code and models are released at https://github.com/lxtGH/Video-K-Net.

Motivation & Objective

• To simplify the complex pipeline of existing video panoptic segmentation methods that rely on separate modules for segmentation and tracking.
• To unify 'things' and 'stuff' segmentation with end-to-end instance-level tracking in a single framework.
• To reduce computational overhead and eliminate the need for auxiliary components like RoI heads or tracking queries.
• To improve temporal consistency in video segmentation through kernel-level fusion and association.
• To establish a strong, simple, and generalizable baseline for unified video segmentation.

Proposed method

• Leverages learnable kernels from K-Net as intrinsic representations for both appearance and context encoding across video frames.
• Introduces a kernel association head with a contrastive learning loss to generate discriminative embeddings for temporal tracking.
• Applies a temporal kernel fusion module to fuse kernels across adjacent frames at the kernel level, avoiding expensive feature-level attention or cascaded modules.
• Uses Hungarian assignment to match kernels with ground truth masks during training, ensuring robustness and reducing noise from unmatched samples.
• Performs online tracking by linking kernels across frames based on learned embeddings, eliminating the need for separate tracking queries or post-processing NMS.
• Employs joint training on key and reference frames to improve kernel grouping and separation for consistent segmentation and tracking.

Experimental results

Research questions

• RQ1 Can learnable kernels from image segmentation models like K-Net naturally support end-to-end video instance tracking without additional tracking heads or queries?
• RQ2 How does kernel-level temporal fusion compare to feature-level fusion in terms of accuracy and efficiency for video panoptic segmentation?
• RQ3 What is the impact of training strategy—specifically, using key and reference frames jointly—on kernel association and segmentation performance?
• RQ4 How does the proposed kernel-based association method compare to RoI-based or query-based tracking in terms of accuracy and computational cost?
• RQ5 To what extent can a unified kernel-based framework generalize across different video segmentation tasks, including semantic and instance segmentation?

Key findings

• Video K-Net achieves a 12% relative improvement on KITTI-STEP, significantly outperforming previous methods with a simpler pipeline.
• On VIPSeg, the method achieves 39.8% VPQ, boosting previous results by nearly 15% relative improvement.
• The framework improves baseline K-Net by 3.5% in STQ on KITTI-STEP while adding only 2.3% GFLOPs and 1.8% parameters.
• On VSPW for video semantic segmentation, Video K-Net boosts various baselines by 2% in mIoU, demonstrating strong generalization.
• For video instance segmentation on YouTube-2019, it achieves 40.5% mAP with ResNet-50 and 54.1% with Swin-Base, showing strong performance on a large-scale benchmark.
• Ablation studies confirm that linking kernels at the final stage and using matched kernel samples during training yield the best results, while early fusion degrades performance.