[Paper Review] CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentation

Monika Wysoczańska, Oriane Siméoni+4 more|arXiv (Cornell University)|Dec 19, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

CLIP-DINOiser enhances CLIP's zero-shot open-vocabulary semantic segmentation by refining its dense features using self-supervised localization priors from DINO, via two lightweight convolutional layers applied in a single forward pass of CLIP. The method achieves state-of-the-art performance on COCO, Pascal Context, Cityscapes, and ADE20k without any annotations or retraining.

ABSTRACT

The popular CLIP model displays impressive zero-shot capabilities thanks to its seamless interaction with arbitrary text prompts. However, its lack of spatial awareness makes it unsuitable for dense computer vision tasks, e.g., semantic segmentation, without an additional fine-tuning step that often uses annotations and can potentially suppress its original open-vocabulary properties. Meanwhile, self-supervised representation methods have demonstrated good localization properties without human-made annotations nor explicit supervision. In this work, we take the best of both worlds and propose an open-vocabulary semantic segmentation method, which does not require any annotations. We propose to locally improve dense MaskCLIP features, which are computed with a simple modification of CLIP's last pooling layer, by integrating localization priors extracted from self-supervised features. By doing so, we greatly improve the performance of MaskCLIP and produce smooth outputs. Moreover, we show that the used self-supervised feature properties can directly be learnt from CLIP features. Our method CLIP-DINOiser needs only a single forward pass of CLIP and two light convolutional layers at inference, no extra supervision nor extra memory and reaches state-of-the-art results on challenging and fine-grained benchmarks such as COCO, Pascal Context, Cityscapes and ADE20k. The code to reproduce our results is available at https://github.com/wysoczanska/clip_dinoiser.

Motivation & Objective

• To enable zero-shot open-vocabulary semantic segmentation using CLIP without requiring any human-annotated segmentation data.
• To improve the spatial localization quality of CLIP’s dense features, which are inherently noisy and lack fine-grained structure.
• To preserve CLIP’s original open-vocabulary capabilities by avoiding fine-tuning or class-specific prototype construction.
• To achieve high performance with minimal computational overhead, using only a single CLIP forward pass and two lightweight convolutional layers.
• To demonstrate that CLIP already encodes useful localization priors that can be extracted and enhanced using self-supervised guidance.

Proposed method

• Adapts MaskCLIP by modifying its global self-attention layer into a convolutional layer to produce dense patch-level features while preserving vision-language alignment.
• Introduces a learnable convolutional pooling layer that mimics DINO’s patch correlation patterns to produce concept-aware, localized feature pooling from CLIP features.
• Leverages self-supervised DINO features as a distillation target to guide the learning of pooling weights without requiring any annotations.
• Introduces a second lightweight convolutional layer to learn objectness scores from CLIP features, emulating the unsupervised objectness map (FOUND) from DINO.
• Uses the learned pooling and objectness maps to generate high-quality segmentation masks in a single forward pass of CLIP.
• Employs a dual-path mechanism: one for concept-aware localization and one for background detection, both trained using DINO as a guide.

Experimental results

Research questions

• RQ1 Can CLIP’s dense features be effectively refined to improve localization without fine-tuning or annotations?
• RQ2 To what extent can self-supervised DINO features guide CLIP to produce better-aligned, smoother segmentation masks?
• RQ3 Can the objectness and patch-correlation priors from DINO be directly transferred to CLIP features using only lightweight learnable layers?
• RQ4 Does a single-pass inference pipeline with minimal added computation outperform multi-pass or prototype-based methods in zero-shot semantic segmentation?
• RQ5 Can CLIP-DINOiser maintain open-vocabulary flexibility while achieving state-of-the-art performance on fine-grained and complex benchmarks?

Key findings

• CLIP-DINOiser achieves state-of-the-art performance on COCO, Pascal Context, Cityscapes, and ADE20k with +6.7 mIoU, +5.1 mIoU, +5.0 mIoU, and +2.2 mIoU improvements over the second-best method, respectively.
• On Pascal VOC, the method achieves 60.1 mIoU, improving by +6.4 mIoU over MaskCLIP, and up to 62.1 mIoU when using an uncertainty-based refinement of the FOUND objectness map.
• The method outperforms OVDiff by +2.3 mIoU on Context and +0.2 mIoU on Object, despite OVDiff requiring per-concept prototype construction.
• CLIP-DINOiser produces more accurate and fine-grained localization than CLIP-DIY and TCL, especially on complex scenes like Cityscapes and ADE20k.
• The method maintains full open-vocabulary capability and operates in a single forward pass of CLIP, with only two additional lightweight convolutional layers at inference.
• The ablation study confirms that the performance gain comes from both the concept-aware pooling and the learned objectness map, with the latter significantly improving background segmentation.