[論文レビュー] DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting

Yongming Rao, Wenliang Zhao+6 more|arXiv (Cornell University)|Dec 2, 2021

Multimodal Machine Learning Applications参考文献 51被引用数 37

ひとこと要約

DenseCLIP は vision-language pre-training (CLIP) から dense prediction へ知識を転送する。画像-テキスト一致をピクセル-テキスト一致へ変換し、トレーニング中に文脈を意識した prompting を適用して言語埋め込みを洗練させる。

ABSTRACT

Recent progress has shown that large-scale pre-training using contrastive image-text pairs can be a promising alternative for high-quality visual representation learning from natural language supervision. Benefiting from a broader source of supervision, this new paradigm exhibits impressive transferability to downstream classification tasks and datasets. However, the problem of transferring the knowledge learned from image-text pairs to more complex dense prediction tasks has barely been visited. In this work, we present a new framework for dense prediction by implicitly and explicitly leveraging the pre-trained knowledge from CLIP. Specifically, we convert the original image-text matching problem in CLIP to a pixel-text matching problem and use the pixel-text score maps to guide the learning of dense prediction models. By further using the contextual information from the image to prompt the language model, we are able to facilitate our model to better exploit the pre-trained knowledge. Our method is model-agnostic, which can be applied to arbitrary dense prediction systems and various pre-trained visual backbones including both CLIP models and ImageNet pre-trained models. Extensive experiments demonstrate the superior performance of our methods on semantic segmentation, object detection, and instance segmentation tasks. Code is available at https://github.com/raoyongming/DenseCLIP

研究の動機と目的

• Bridge the gap between image-text pre-training (CLIP) and dense prediction tasks (per-pixel outputs).
• Leverage pixel-text matching to guide dense predictors and use image context to prompt the language model.
• Demonstrate model-agnostic applicability to various backbones and dense prediction frameworks.
• Show improvements on semantic segmentation, object detection, and instance segmentation over ImageNet-pretrained baselines and vanilla CLIP fine-tuning.

提案手法

• Convert image-text matching in CLIP to pixel-text matching using a language-compatible feature map from the CLIP image encoder and a text embedding set for K classes.
• Compute pixel-text score maps s by normalizing and multiplying the language-compatible features z with text features t: s = z_hat t_hat^T.
• Use the score maps as auxiliary segmentation loss and concatenate them with image features to guide dense predictors: x_4' = [x_4, s].
• Explore context-aware prompting by (a) pre-model prompting using visual context to produce refined text inputs, and (b) post-model prompting refining text features after the text encoder via a Transformer decoder; select post-model prompting as default for efficiency and performance.
• Auxiliary losses: (i) segmentation loss on the pixel-text score maps for segmentation tasks, (ii) binary cross-entropy loss on pixel-text maps for detection/instance segmentation.
• Show compatibility with backbones beyond CLIP image encoders (e.g., ImageNet-pretrained ResNet, Swin) by freezing the CLIP text encoder and guiding the backbone with language priors.]
• research_questions: ["Can CLIP-based language priors improve dense prediction tasks beyond classification?","How can pixel-text alignment be leveraged to guide dense prediction models effectively?","Do context-aware prompting strategies improve the transfer of vision-language knowledge to dense tasks?","Is DenseCLIP backbones-agnostic and can it benefit non-CLIP image encoders?","What are the empirical gains on semantic segmentation, object detection, and instance segmentation when applying DenseCLIP?"]
• key_findings:[

実験結果

リサーチクエスチョン

• RQ1 Can CLIP-based language priors improve dense prediction tasks beyond classification?
• RQ2 How can pixel-text alignment be leveraged to guide dense prediction models effectively?
• RQ3 Do context-aware prompting strategies improve the transfer of vision-language knowledge to dense tasks?
• RQ4 Is DenseCLIP backbones-agnostic and can it benefit non-CLIP image encoders?
• RQ5 What are the empirical gains on semantic segmentation, object detection, and instance segmentation when applying DenseCLIP?

主な発見

• DenseCLIP は ADE20K の semantic segmentation を改善し、ResNet-50、ResNet-101、ViT-B などのバックボーンで ImageNet pre-trained baselines より高い mIoU を達成。
• DenseCLIP は ADE20K で Semantic FPN と組み合わせた場合に vanilla CLIP fine-tuning より顕著な利益を生む（例：ResNet-50: 43.5 vs 39.6 mIoU SS）。
• COCO では DenseCLIP は ImageNet pre-training および vanilla CLIP fine-tuning を上回るオブジェクト検出およびインスタンスセグメンテーションの改善を一貫して示し、AP とマスクAP の向上を RetinaNet と Mask R-CNN の組み合わせで示す（例：RN50-DenseCLIP vs RN50-CLIP の AP）。
• Post-model prompting（テキストエンコーダの後での言語 refinements）は pre-model prompting より少ない追加 FLOPs でより良い性能を提供。
• DenseCLIP は DenseCLIP ガイダンスと組み合わせることで non-CLIP バックボーン（ResNet, Swin）も改善可能で、フレームワークの汎用性を示す。
• 単一の ResNet-101 と DenseCLIP、軽量デコーダで競争力のある ADE20K の性能を、いくつかの最先端手法と比較して計算量を抑えつつ達成。