[Paper Review] End-to-End Video Instance Segmentation with Transformers

Yuqing Wang, Zhaoliang Xu+5 more|arXiv (Cornell University)|Nov 30, 2020

Video Analysis and SummarizationComputer Science29 references66 citations

TL;DR

VisTR treats video instance segmentation as end-to-end parallel sequence decoding with Transformers, producing per-instance mask sequences across frames and achieving fast, competitive results with a single model.

ABSTRACT

Video instance segmentation (VIS) is the task that requires simultaneously classifying, segmenting and tracking object instances of interest in video. Recent methods typically develop sophisticated pipelines to tackle this task. Here, we propose a new video instance segmentation framework built upon Transformers, termed VisTR, which views the VIS task as a direct end-to-end parallel sequence decoding/prediction problem. Given a video clip consisting of multiple image frames as input, VisTR outputs the sequence of masks for each instance in the video in order directly. At the core is a new, effective instance sequence matching and segmentation strategy, which supervises and segments instances at the sequence level as a whole. VisTR frames the instance segmentation and tracking in the same perspective of similarity learning, thus considerably simplifying the overall pipeline and is significantly different from existing approaches. Without bells and whistles, VisTR achieves the highest speed among all existing VIS models, and achieves the best result among methods using single model on the YouTube-VIS dataset. For the first time, we demonstrate a much simpler and faster video instance segmentation framework built upon Transformers, achieving competitive accuracy. We hope that VisTR can motivate future research for more video understanding tasks.

Motivation & Objective

• Motivate a simple, end-to-end VIS framework that avoids multi-stage pipelines.
• Leverage Transformers to model temporal and spatial relationships across video frames.
• Develop an instance sequence matching and segmentation strategy suited for sequence-level supervision.
• Demonstrate strong accuracy while maintaining high inference speed on YouTube-VIS.

Proposed method

• Encode multi-frame video clips with a CNN backbone to obtain clip-level features.
• Use a Transformer encoder to capture pixel-level cross-frame relationships with 3D positional encodings.
• Decode a fixed number of instance predictions with a Transformer decoder using instance queries.
• Apply instance sequence matching to align predicted and ground-truth instance sequences across frames via Hungarian algorithm.
• Perform instance sequence segmentation by accumulating mask features over time and predicting mask sequences with 3D convolutions.
• Train with a Hungarian loss that combines classification, box sequence, and mask sequence losses.

Figure 1 : The overall architecture of VisTR. It contains four main components: 1) a CNN backbone that extracts feature representation of multiple images; 2) an encoder-decoder Transformer that models the relations of pixel-level features and decodes the instance-level features; 3) an instance seque

Experimental results

Research questions

• RQ1 Can a Transformer-based framework directly predict temporally-consistent instance masks for entire video clips?
• RQ2 Does end-to-end sequence prediction with instance-aware queries simplify VIS while maintaining accuracy and speed?
• RQ3 How does sequence-level supervision via instance sequence matching affect training and tracking in VIS?
• RQ4 What is the impact of temporal information, positional encoding, and 3D segmentation on VIS performance?

Key findings

• VisTR achieves 40.1% AP on YouTube-VIS validation with ResNet-101 and 57.7 FPS (or 27.7 FPS excluding data loading).
• With ResNet-50, VisTR attains 36.2 AP and 30.0 FPS (69.9 with data loading).
• VisTR outperforms several single-model VIS methods in AP on the YouTube-VIS val set.
• The ablations show that longer video sequences and explicit positional encoding significantly improve performance (AP up to 33.3).
• Instance-level queries can closely match the prediction-level configuration while reducing parameter count by sharing embeddings across frames.
• Incorporating an instance sequence segmentation module with 3D convolutions yields additional AP gains (≈1.1 points).

Figure 2 : Visualization of VisTR on the YouTube-VIS [ 30 ] validation dataset. Each row contains images from the same video. For each video, here the same colors depict the mask sequences of the same instances (Best viewed on screen).