[Paper Review] OpenVIS: Open-vocabulary Video Instance Segmentation

Pinxue Guo, Tony Jun Huang+5 more|arXiv (Cornell University)|May 26, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper introduces OpenVIS, a novel task for open-vocabulary video instance segmentation that enables zero-shot detection, segmentation, and tracking of any object via text prompts, regardless of training category coverage. It proposes a two-stage framework using a binary-loss-trained mask proposal network and a SquareCrop post-processing strategy to enhance compatibility with pre-trained vision-language models like CLIP, achieving a 148% improvement over full-supervised baselines on BURST.

ABSTRACT

Open-vocabulary Video Instance Segmentation (OpenVIS) can simultaneously detect, segment, and track arbitrary object categories in a video, without being constrained to categories seen during training. In this work, we propose InstFormer, a carefully designed framework for the OpenVIS task that achieves powerful open-vocabulary capabilities through lightweight fine-tuning with limited-category data. InstFormer begins with the open-world mask proposal network, encouraged to propose all potential instance class-agnostic masks by the contrastive instance margin loss. Next, we introduce InstCLIP, adapted from pre-trained CLIP with Instance Guidance Attention, which encodes open-vocabulary instance tokens efficiently. These instance tokens not only enable open-vocabulary classification but also offer strong universal tracking capabilities. Furthermore, to prevent the tracking module from being constrained by the training data with limited categories, we propose the universal rollout association, which transforms the tracking problem into predicting the next frame's instance tracking token. The experimental results demonstrate the proposed InstFormer achieve state-of-the-art capabilities on a comprehensive OpenVIS evaluation benchmark, while also achieves competitive performance in fully supervised VIS task.

Motivation & Objective

• To address the limitation of existing video instance segmentation models that can only recognize objects from fixed, closed-set categories in their training data.
• To enable zero-shot segmentation of any object described by a text prompt, even if the category was not seen during training.
• To develop a robust evaluation benchmark for open-vocabulary video instance segmentation using existing datasets to measure zero-shot generalization capability.
• To improve compatibility between masked object proposals and pre-trained vision-language models (VLMs) like CLIP, which are sensitive to input distortion and aspect ratio.
• To explore the effectiveness of different backbone architectures and training data regimes in maximizing open-vocabulary performance.

Proposed method

• Replace the classification head in a query-based mask proposal network with an instance head trained using a binary cross-entropy loss to encourage proposal of all potential objects, regardless of category.
• Apply a novel SquareCrop post-processing strategy that crops masked proposals to a square aspect ratio while preserving object content and minimizing distortion, improving compatibility with CLIP.
• Use a pre-trained vision-language model (CLIP) to compute visual-textual similarity between each proposal and a set of candidate text descriptions, assigning the most similar category.
• Train the mask proposal network on a combination of COCO and YouTube-VIS datasets with mask annotations but without category labels (C+Y’), enabling generalization to unseen classes.
• Fine-tune the CLIP visual encoder with a contrastive learning objective on masked proposals to further align visual representations with the downstream segmentation task.
• Leverage the UVO dataset with only mask annotations (C+U’) for additional pre-training to improve mask proposal quality on rare and unseen categories.

Experimental results

Research questions

• RQ1 Can a video instance segmentation model be effectively extended to zero-shot open-vocabulary recognition using only text prompts, without requiring category annotations during training?
• RQ2 How can mask proposal networks be optimized to generate high-quality, class-agnostic masks for all potential objects in a video, even those outside the training category set?
• RQ3 What post-processing strategy for masked proposals maximizes zero-shot classification accuracy when using pre-trained VLMs like CLIP?
• RQ4 How does the choice of backbone architecture and training data (especially with only mask annotations) affect the performance of open-vocabulary video instance segmentation?
• RQ5 To what extent can the proposed framework generalize to rare and unseen categories not present in the COCO dataset?

Key findings

• The proposed OpenVIS framework achieves a 148% improvement in mean average precision (AP) over full-supervised baselines on the BURST dataset, demonstrating strong zero-shot generalization.
• The SquareCrop post-processing strategy improves CLIP-based classification accuracy by 21% (absolute) compared to direct masked input and 23% compared to resized bounding box crops on the BURST validation set.
• Using the Swin-L backbone with ViT-B CLIP encoder and training on the UVO dataset (C+U’) yields an AP of 4.97 on BURST, indicating strong scalability with larger models and more data.
• On the uncommon 404 categories (not in COCO), the method achieves an AP of 4.15 with the Swin-L backbone, showing strong performance on rare and unseen classes.
• The model achieves an AP of 3.48 using ViT-B as the CLIP visual encoder and 2.56 with ResNet-50, demonstrating consistent gains across different backbone configurations.
• The ablation study confirms that the binary loss in the mask proposal network significantly improves mask coverage and detection of potential instances, especially for rare categories.