[Paper Review] DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting

Yongming Rao, Wenliang Zhao+6 more|arXiv (Cornell University)|Dec 2, 2021

Multimodal Machine Learning Applications51 references37 citations

TL;DR

DenseCLIP transfers knowledge from vision-language pre-training (CLIP) to dense prediction by converting image-text matching to pixel-text matching and applying context-aware prompting to refine language embeddings during training.

ABSTRACT

Recent progress has shown that large-scale pre-training using contrastive image-text pairs can be a promising alternative for high-quality visual representation learning from natural language supervision. Benefiting from a broader source of supervision, this new paradigm exhibits impressive transferability to downstream classification tasks and datasets. However, the problem of transferring the knowledge learned from image-text pairs to more complex dense prediction tasks has barely been visited. In this work, we present a new framework for dense prediction by implicitly and explicitly leveraging the pre-trained knowledge from CLIP. Specifically, we convert the original image-text matching problem in CLIP to a pixel-text matching problem and use the pixel-text score maps to guide the learning of dense prediction models. By further using the contextual information from the image to prompt the language model, we are able to facilitate our model to better exploit the pre-trained knowledge. Our method is model-agnostic, which can be applied to arbitrary dense prediction systems and various pre-trained visual backbones including both CLIP models and ImageNet pre-trained models. Extensive experiments demonstrate the superior performance of our methods on semantic segmentation, object detection, and instance segmentation tasks. Code is available at https://github.com/raoyongming/DenseCLIP

Motivation & Objective

• Bridge the gap between image-text pre-training (CLIP) and dense prediction tasks (per-pixel outputs).
• Leverage pixel-text matching to guide dense predictors and use image context to prompt the language model.
• Demonstrate model-agnostic applicability to various backbones and dense prediction frameworks.
• Show improvements on semantic segmentation, object detection, and instance segmentation over ImageNet-pretrained baselines and vanilla CLIP fine-tuning.

Proposed method

• Convert image-text matching in CLIP to pixel-text matching using a language-compatible feature map from the CLIP image encoder and a text embedding set for K classes.
• Compute pixel-text score maps s by normalizing and multiplying the language-compatible features z with text features t: s = z_hat t_hat^T.
• Use the score maps as auxiliary segmentation loss and concatenate them with image features to guide dense predictors: x_4' = [x_4, s].
• Explore context-aware prompting by (a) pre-model prompting using visual context to produce refined text inputs, and (b) post-model prompting refining text features after the text encoder via a Transformer decoder; select post-model prompting as default for efficiency and performance.
• Auxiliary losses: (i) segmentation loss on the pixel-text score maps for segmentation tasks, (ii) binary cross-entropy loss on pixel-text maps for detection/instance segmentation.
• Show compatibility with backbones beyond CLIP image encoders (e.g., ImageNet-pretrained ResNet, Swin) by freezing the CLIP text encoder and guiding the backbone with language priors.

Experimental results

Research questions

• RQ1 Can CLIP-based language priors improve dense prediction tasks beyond classification?
• RQ2 How can pixel-text alignment be leveraged to guide dense prediction models effectively?
• RQ3 Do context-aware prompting strategies improve the transfer of vision-language knowledge to dense tasks?
• RQ4 Is DenseCLIP backbones-agnostic and can it benefit non-CLIP image encoders?
• RQ5 What are the empirical gains on semantic segmentation, object detection, and instance segmentation when applying DenseCLIP?

Key findings

• DenseCLIP improves semantic segmentation on ADE20K, achieving higher mIoU than ImageNet pre-trained baselines across backbones (e.g., ResNet-50, ResNet-101, ViT-B).
• DenseCLIP yields notable gains over vanilla CLIP fine-tuning when paired with Semantic FPN on ADE20K (e.g., ResNet-50: 43.5 vs 39.6 mIoU SS).
• On COCO, DenseCLIP consistently boosts object detection and instance segmentation over ImageNet pre-training and vanilla CLIP fine-tuning, with gains in AP and mask AP across RetinaNet and Mask R-CNN (e.g., RN50-DenseCLIP vs RN50-CLIP for AP).
• Post-model prompting (language refinement after the text encoder) provides better performance with fewer extra FLOPs than pre-model prompting.
• DenseCLIP can improve non-CLIP backbones (ResNet, Swin) when paired with DenseCLIP guidance, showing the framework’s generality.
• A single ResNet-101 with DenseCLIP and a lightweight decoder achieves competitive ADE20K performance with reduced computation compared to some state-of-the-art methods.