[Paper Review] CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation

Seokju Cho, Heeseong Shin+4 more|arXiv (Cornell University)|Mar 21, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

CAT-Seg proposes a novel cost aggregation framework for open-vocabulary semantic segmentation that optimizes the CLIP image-text similarity map (cost volume) instead of fine-tuning CLIP embeddings directly, preserving open-vocabulary capabilities. By using a Transformer-based module to jointly aggregate spatial and class-wise relations in the cost volume with embedding guidance, CAT-Seg achieves state-of-the-art performance across ADE20K, PASCAL VOC, and PASCAL-Context, with a 22% mIoU improvement over prior methods on ADE20K-847.

ABSTRACT

Open-vocabulary semantic segmentation presents the challenge of labeling each pixel within an image based on a wide range of text descriptions. In this work, we introduce a novel cost-based approach to adapt vision-language foundation models, notably CLIP, for the intricate task of semantic segmentation. Through aggregating the cosine similarity score, i.e., the cost volume between image and text embeddings, our method potently adapts CLIP for segmenting seen and unseen classes by fine-tuning its encoders, addressing the challenges faced by existing methods in handling unseen classes. Building upon this, we explore methods to effectively aggregate the cost volume considering its multi-modal nature of being established between image and text embeddings. Furthermore, we examine various methods for efficiently fine-tuning CLIP.

Motivation & Objective

• Address the challenge of transferring CLIP's image-level vision-language knowledge to pixel-level segmentation without degrading its open-vocabulary capabilities.
• Overcome limitations of two-stage methods that rely on biased region proposals and lack global context understanding.
• Develop a method that enables effective, generalizable segmentation for arbitrary unseen categories at inference time.
• Preserve CLIP's pre-trained open-vocabulary generalization by optimizing the cost volume instead of fine-tuning image embeddings directly.
• Enable robust part-level segmentation for fine-grained, unseen categories, overcoming biases in existing two-stage approaches.

Proposed method

• Construct a dense cost volume as the cosine similarity map between CLIP's image and text embeddings.
• Design a Transformer-based cost aggregation module that decomposes aggregation into spatial and class-wise components.
• Introduce embedding guidance to enhance the alignment between visual and textual features during cost aggregation.
• Use a lightweight decoder to reconstruct segmentation masks from the aggregated cost volume while preserving fine details.
• Fine-tune only the attention layers in the Transformer, avoiding full fine-tuning of CLIP to preserve pre-trained knowledge.
• Train end-to-end using the cost volume as the primary optimization target, minimizing degradation of CLIP’s open-vocabulary performance.

Figure 1: Teaser. For open-vocabulary semantic segmentation, we propose to aggregate a matching cost derived from dense image and text embeddings of CLIP [ 44 ] , resulting in state-of-the-art performance across all benchmarks.

Experimental results

Research questions

• RQ1 Can optimizing the CLIP image-text similarity map (cost volume) instead of the image embeddings preserve CLIP’s open-vocabulary generalization in pixel-level segmentation?
• RQ2 How does cost aggregation with spatial and class-wise attention improve segmentation performance for both seen and unseen categories?
• RQ3 Does the proposed method generalize better than two-stage approaches that use region proposals and independently classify each region?
• RQ4 Can the framework handle fine-grained, unseen categories such as 'arm' or 'birdcage' without prior exposure during training?
• RQ5 How does the model scale with increasing backbone capacity, and does it maintain efficiency and performance gains?

Key findings

• CAT-Seg achieves state-of-the-art performance on all benchmarks, including a 22% mIoU improvement over prior methods on ADE20K-847.
• On ADE20K-847, the model achieves an mIoU of 44.9 with ViT-G backbone, showing strong performance on large-scale open-vocabulary segmentation.
• The method shows a 9.0% mIoU gain when scaling from ViT-B to ViT-G on ADE20K-150, indicating favorable scaling behavior.
• Inference speed is faster than the two-stage baseline, especially as model size increases, due to the cost-volume-based computation that scales with number of classes.
• The model generalizes well to part segmentation: it successfully segments unseen fine-grained categories like 'arm' and 'leg', while two-stage methods fail due to proposal bias.
• Ablation studies confirm that optimizing the cost volume prevents performance degradation in open-vocabulary capabilities, unlike direct fine-tuning of CLIP embeddings.

(c) Qualitative comparisons