[Paper Review] Language-driven Semantic Segmentation

Boyi Li, Kilian Q. Weinberger+3 more|arXiv (Cornell University)|Jan 10, 2022

Advanced Neural Network Applications163 citations

TL;DR

LSeg uses a text encoder (e.g., CLIP) to embed arbitrary label descriptions and trains a dense image encoder to align per-pixel embeddings with these text embeddings, enabling zero-shot semantic segmentation and flexible label sets without retraining.

ABSTRACT

We present LSeg, a novel model for language-driven semantic image segmentation. LSeg uses a text encoder to compute embeddings of descriptive input labels (e.g., "grass" or "building") together with a transformer-based image encoder that computes dense per-pixel embeddings of the input image. The image encoder is trained with a contrastive objective to align pixel embeddings to the text embedding of the corresponding semantic class. The text embeddings provide a flexible label representation in which semantically similar labels map to similar regions in the embedding space (e.g., "cat" and "furry"). This allows LSeg to generalize to previously unseen categories at test time, without retraining or even requiring a single additional training sample. We demonstrate that our approach achieves highly competitive zero-shot performance compared to existing zero- and few-shot semantic segmentation methods, and even matches the accuracy of traditional segmentation algorithms when a fixed label set is provided. Code and demo are available at https://github.com/isl-org/lang-seg.

Motivation & Objective

• Address the limitation of fixed label sets in semantic segmentation by enabling flexible, language-driven label representations.
• Leverage a text encoder to embed descriptive labels and train an image encoder to align pixel embeddings with these label embeddings.
• Demonstrate zero-shot and few-shot segmentation capabilities without additional training samples.
• Show that semantic similarity in the language space transfers to the visual domain for unseen classes.

Proposed method

• Embed labels with a pretrained text encoder (CLIP) to produce a set of label embeddings regardless of their number or order.
• Use a dense prediction transformer image encoder to produce per-pixel embeddings for the input image.
• Compute a pixel-wise correlation tensor by inner products between pixel embeddings and label embeddings, and train with a pixel-wise softmax cross-entropy loss to align ground-truth pixels with their labels.
• Incorporate a spatial regularization block (DepthwiseBlock or BottleneckBlock) to upsample and refine predictions while preserving label order equivariance.
• Freeze the text encoder during training and update only the image encoder, enabling flexible synthesis of zero-shot segmentation maps for arbitrary label sets.

Experimental results

Research questions

• RQ1 Can a language-embedded label space enable accurate zero-shot semantic segmentation without retraining for new classes?
• RQ2 How does replacing or expanding the label set at test time affect segmentation quality and flexibility?
• RQ3 To what extent do language-driven label embeddings align semantically related concepts (e.g., dog and pet) in guiding pixel labeling?
• RQ4 What is the impact of different text encoders and backbones on zero-shot segmentation performance?
• RQ5 How does LSeg compare to fixed-label and few-shot segmentation baselines on standard benchmarks?

Key findings

• LSeg achieves competitive zero-shot performance compared to existing zero- and few-shot methods across benchmarks.
• With a larger backbone (ViT-L/16), LSeg attains strong zero-shot results, rivaling some few-shot methods.
• The text embeddings introduce only a small performance drop relative to fixed-label segmentation when labels are fixed.
• LSeg can synthesize zero-shot segmentation models on the fly by changing input label sets without retraining.
• A spatial regularization module improves per-pixel predictions without compromising the label-flexible framework.