[论文解读] End-to-End Video Instance Segmentation with Transformers

Yuqing Wang, Zhaoliang Xu+5 more|arXiv (Cornell University)|Nov 30, 2020

Video Analysis and Summarization参考文献 29被引用 66

一句话总结

VisTR 将视频实例分割视为端到端并行序列解码，使用 Transformers，在跨帧为每个实例生成掩码序列，并以单模型实现快速、具有竞争力的结果。

ABSTRACT

Video instance segmentation (VIS) is the task that requires simultaneously classifying, segmenting and tracking object instances of interest in video. Recent methods typically develop sophisticated pipelines to tackle this task. Here, we propose a new video instance segmentation framework built upon Transformers, termed VisTR, which views the VIS task as a direct end-to-end parallel sequence decoding/prediction problem. Given a video clip consisting of multiple image frames as input, VisTR outputs the sequence of masks for each instance in the video in order directly. At the core is a new, effective instance sequence matching and segmentation strategy, which supervises and segments instances at the sequence level as a whole. VisTR frames the instance segmentation and tracking in the same perspective of similarity learning, thus considerably simplifying the overall pipeline and is significantly different from existing approaches. Without bells and whistles, VisTR achieves the highest speed among all existing VIS models, and achieves the best result among methods using single model on the YouTube-VIS dataset. For the first time, we demonstrate a much simpler and faster video instance segmentation framework built upon Transformers, achieving competitive accuracy. We hope that VisTR can motivate future research for more video understanding tasks.

研究动机与目标

• 提出一个简单的端到端 VIS 框架，避免多阶段管线。
• 利用 Transformer 来建模视频帧之间的时间和空间关系。
• 开发适用于序列级监督的实例序列匹配与分割策略。
• 在 YouTube-VIS 上展示高准确性，同时保持较高的推断速度。

提出的方法

• 使用 CNN 主干对多帧视频片段进行编码，以获得片段级特征。
• 使用 Transformer 编码器通过 3D 位置编码捕获跨帧像素级关系。
• 使用 Transformer 解码器和实例查询解码固定数量的实例预测。
• 通过 Hungarian 算法对齐预测实例序列与地面真值序列在跨帧上的对齐。
• 通过时序累计掩码特征并使用 3D 卷积预测掩码序列，执行实例序列分割。
• 用结合分类、框序列和掩码序列损失的 Hungarian 损失进行训练。

Figure 1 : The overall architecture of VisTR. It contains four main components: 1) a CNN backbone that extracts feature representation of multiple images; 2) an encoder-decoder Transformer that models the relations of pixel-level features and decodes the instance-level features; 3) an instance seque

实验结果

研究问题

• RQ1 一个基于 Transformer 的框架能否直接为整段视频片段预测时间上一致的实例掩码？
• RQ2 端到端的序列预测配合实例感知查询是否在保持准确性和速度的同时简化了 VIS？
• RQ3 通过实例序列匹配的序列级监督对 VIS 的训练和跟踪有何影响？
• RQ4 时间信息、位置编码和 3D 分割对 VIS 性能有何影响？

主要发现

模型	骨干网络	FPS	AP	AP50	AP75	AR1	AR10
DeepSORT	ResNet-50	-	26.1	42.9	26.1	27.8	31.3
FEELVOS	ResNet-50	-	26.9	42.0	29.7	29.9	33.4
OSMN	ResNet-50	-	27.5	45.1	29.1	28.6	33.1
MaskTrack R-CNN	ResNet-50	20.0	30.3	51.1	32.6	31.0	35.5
STEm-Seg	ResNet-50	-	30.6	50.7	33.5	31.6	37.1
STEm-Seg	ResNet-101	2.1	34.6	55.8	37.9	34.4	41.6
MaskProp	ResNet-50	-	40.0	-	-	-	-
MaskProp	ResNet-101	-	42.5	-	-	-	-
VisTR	ResNet-50	30.0/69.9	36.2	59.8	36.9	37.2	42.4
VisTR	ResNet-101	27.7/57.7	40.1	64.0	45.0	38.3	44.9

• VisTR 在 YouTube-VIS 验证集上以 ResNet-101 达到 40.1% AP，FPS 为 57.7（或不含数据加载时为 27.7 FPS）。
• 使用 ResNet-50 时，VisTR 达到 36.2 AP 和 30.0 FPS（包含数据加载后为 69.9）。
• VisTR 在 YouTube-VIS 验证集的 AP 上超越多种单模型 VIS 方法。
• 消融研究显示更长的视频序列和显式位置编码显著提升性能（AP 高达 33.3）。
• 实例级查询可以在接近预测级配置的同时减少通过跨帧共享嵌入来的参数数量。
• 引入带有 3D 卷积的实例序列分割模块可带来额外的 AP 增益（约 1.1 点）。

Figure 2 : Visualization of VisTR on the YouTube-VIS [ 30 ] validation dataset. Each row contains images from the same video. For each video, here the same colors depict the mask sequences of the same instances (Best viewed on screen).