[Paper Review] Open-vocabulary Object Detection via Vision and Language Knowledge Distillation

Xiuye Gu, Tsung-Yi Lin+2 more|arXiv (Cornell University)|Apr 28, 2021

Multimodal Machine Learning Applications39 references280 citations

TL;DR

ViLD distills knowledge from open-vocabulary image classifiers into a two-stage detector to enable open-vocabulary object detection, achieving strong novel-category precision and transferability across datasets.

ABSTRACT

We aim at advancing open-vocabulary object detection, which detects objects described by arbitrary text inputs. The fundamental challenge is the availability of training data. It is costly to further scale up the number of classes contained in existing object detection datasets. To overcome this challenge, we propose ViLD, a training method via Vision and Language knowledge Distillation. Our method distills the knowledge from a pretrained open-vocabulary image classification model (teacher) into a two-stage detector (student). Specifically, we use the teacher model to encode category texts and image regions of object proposals. Then we train a student detector, whose region embeddings of detected boxes are aligned with the text and image embeddings inferred by the teacher. We benchmark on LVIS by holding out all rare categories as novel categories that are not seen during training. ViLD obtains 16.1 mask AP$_r$ with a ResNet-50 backbone, even outperforming the supervised counterpart by 3.8. When trained with a stronger teacher model ALIGN, ViLD achieves 26.3 AP$_r$. The model can directly transfer to other datasets without finetuning, achieving 72.2 AP$_{50}$ on PASCAL VOC, 36.6 AP on COCO and 11.8 AP on Objects365. On COCO, ViLD outperforms the previous state-of-the-art by 4.8 on novel AP and 11.4 on overall AP. Code and demo are open-sourced at https://github.com/tensorflow/tpu/tree/master/models/official/detection/projects/vild.

Motivation & Objective

• Address the challenge of detecting objects described by arbitrary text inputs without abundant detection annotations for novel categories.
• Leverage pretrained open-vocabulary image classifiers as teachers to supervise a two-stage detector.
• Develop ViLD components (ViLD-text and ViLD-image) to align region embeddings with text and image embeddings.
• Demonstrate open-vocabulary detection performance on LVIS and transferability to other detection datasets.

Proposed method

• Replace standard classifier in a two-stage detector with text embeddings from a pretrained open-vocabulary model to form ViLD-text.
• Distill image embeddings from a pretrained image encoder into region embeddings of a Mask R-CNN to form ViLD-image using an L1 loss.
• Combine ViLD-text and ViLD-image via a joint training objective L_ViLD = L_ViLD-text + w * L_ViLD-image.
• During inference, use the same text embeddings for base and novel categories to enable open-vocabulary detection (C_B ∪ C_N).
• Optionally apply model ensembling (ViLD-ensemble or ViLD-text+CLIP) to improve base/novel category performance.
• Demonstrate distillation with different teacher models (CLIP, ALIGN) and transferability without fine-tuning.

Experimental results

Research questions

• RQ1 Can knowledge distillation from open-vocabulary image classifiers enable effective open-vocabulary object detection?
• RQ2 How do text-based and image-based distillation signals complement each other for novel category detection?
• RQ3 What is the impact of stronger teacher models (e.g., ALIGN) on open-vocabulary detection performance?
• RQ4 How well does a ViLD-trained detector transfer to other detection datasets without fine-tuning?

Key findings

• ViLD achieves 16.1 novel-category AP (AP_r) on LVIS with a ResNet-50 backbone, surpassing the supervised counterpart by 3.8 AP_r.
• Using a stronger teacher model ALIGN yields 26.3 AP_r on LVIS for novel categories.
• ViLD directly transfers to PASCAL VOC (72.2 AP50), COCO (36.6 AP), and Objects365 (11.8 AP) without finetuning.
• ViLD outperforms the prior open-vocabulary detector on COCO by 4.8 AP_r and 11.4 AP overall.
• ViLD-text (with CLIP text embeddings) significantly improves novel-category AP_r compared to GloVe (10.1 vs 3.0).
• ViLD benefits from combining text-based and image-based distillation (ViLD-text + ViLD-image) to boost novel-category performance.