[Paper Review] Efficient Video Instance Segmentation via Tracklet Query and Proposal

Jialian Wu, Sudhir Yarram+5 more|arXiv (Cornell University)|Mar 3, 2022

Video Analysis and Summarization4 citations

TL;DR

This paper proposes EfficientVIS, a fully end-to-end video instance segmentation framework that uses tracklet queries and proposals to enable efficient, real-time inference and fast training. By leveraging ROI-wise query-video interaction and correspondence learning for inter-clip tracking, it achieves state-of-the-art accuracy on YouTube-VIS with 15× fewer training epochs than VisTR, eliminating the need for hand-crafted data association.

ABSTRACT

Video Instance Segmentation (VIS) aims to simultaneously classify, segment, and track multiple object instances in videos. Recent clip-level VIS takes a short video clip as input each time showing stronger performance than frame-level VIS (tracking-by-segmentation), as more temporal context from multiple frames is utilized. Yet, most clip-level methods are neither end-to-end learnable nor real-time. These limitations are addressed by the recent VIS transformer (VisTR) which performs VIS end-to-end within a clip. However, VisTR suffers from long training time due to its frame-wise dense attention. In addition, VisTR is not fully end-to-end learnable in multiple video clips as it requires a hand-crafted data association to link instance tracklets between successive clips. This paper proposes EfficientVIS, a fully end-to-end framework with efficient training and inference. At the core are tracklet query and tracklet proposal that associate and segment regions-of-interest (RoIs) across space and time by an iterative query-video interaction. We further propose a correspondence learning that makes tracklets linking between clips end-to-end learnable. Compared to VisTR, EfficientVIS requires 15x fewer training epochs while achieving state-of-the-art accuracy on the YouTube-VIS benchmark. Meanwhile, our method enables whole video instance segmentation in a single end-to-end pass without data association at all.

Motivation & Objective

• To address the slow convergence and non-end-to-end nature of existing clip-level video instance segmentation methods.
• To eliminate the need for hand-crafted data association in multi-clip video processing.
• To enable real-time inference and fast training while maintaining high segmentation accuracy.
• To improve robustness to object occlusion and motion blur through enhanced temporal context modeling.
• To achieve fully end-to-end learning across entire videos, including inter-clip tracklet linking.

Proposed method

• Introduces tracklet query, a learnable latent embedding that encodes instance appearance and is updated across clips via correspondence learning.
• Employs tracklet proposal, a space-time tube that defines the region of interest for each instance, reducing computation and background clutter.
• Uses temporal dynamic convolution to enrich temporal context by aggregating features across frames within the proposal.
• Applies factorized temporo-spatial self-attention to allow queries to interact across space and time, enabling full tracklet mask generation in one forward pass.
• Designs a correspondence learning mechanism that shares and updates tracklet queries across clips, enabling end-to-end inter-clip association without data association.
• Adopts an ROI-wise query-video interaction that focuses on proposal-defined regions, drastically reducing redundant attention compared to frame-wise dense attention in VisTR.

Experimental results

Research questions

• RQ1 Can a clip-level video instance segmentation framework achieve fast convergence while maintaining high accuracy?
• RQ2 Can end-to-end learning be extended beyond a single clip to enable seamless, learnable tracklet linking across multiple clips?
• RQ3 Can the reliance on hand-crafted data association in multi-clip VIS be eliminated through a learnable correspondence mechanism?
• RQ4 How does ROI-wise query interaction compare to dense attention in terms of training efficiency and segmentation precision?
• RQ5 Can the framework maintain strong performance under low frame rate conditions where motion-based trackers fail?

Key findings

• EfficientVIS achieves state-of-the-art performance on YouTube-VIS 2019 and 2021 benchmarks without using extra data or multi-model ensembling.
• The method reduces training time by 15× compared to VisTR, achieving convergence in significantly fewer epochs.
• EfficientVIS improves AP75 by 1.5% and AR (recall) by 3.2% over VisTR, indicating better mask precision and detection of occluded objects.
• The framework maintains strong performance at 0.2 FPS (one frame every 5 seconds), demonstrating robustness to low frame rate and motion blur.
• Correspondence learning outperforms hand-crafted data association in multi-clip settings, with a 2.1% higher AP on YouTube-VIS 2019.
• The ROI-wise design reduces background interference and improves mask quality, contributing to higher accuracy despite faster convergence.