[Paper Review] CLIP Is Also a Good Teacher: A New Learning Framework for Inductive Zero-shot Semantic Segmentation

Jialei Chen, Daisuke Deguchi+3 more|arXiv (Cornell University)|Oct 3, 2023

Topic Modeling4 citations

TL;DR

This paper proposes CLIPTeacher, a novel framework for inductive zero-shot semantic segmentation that leverages CLIP as a knowledge distillation teacher without modifying its architecture. It introduces a Global Learning Module (GLM) to align CLIP's CLS token with dense features and a Pixel Learning Module (PLM) to generate pseudo-annotations for unannotated (ignored) regions using multi-scale K-Means on CLIP's dense tokens, enabling full utilization of image content. The method achieves state-of-the-art performance, improving mIoU by 8.8% on PASCAL Context, 2.2% on VOC 2012, and 1.3% on COCO-Stuff 164k.

ABSTRACT

Generalized Zero-shot Semantic Segmentation aims to segment both seen and unseen categories only under the supervision of the seen ones. To tackle this, existing methods adopt the large-scale Vision Language Models (VLMs) which obtain outstanding zero-shot performance. However, as the VLMs are designed for classification tasks, directly adapting the VLMs may lead to sub-optimal performance. Consequently, we propose CLIP-ZSS (Zero-shot Semantic Segmentation), a simple but effective training framework that enables any image encoder designed for closed-set segmentation applied in zero-shot and open-vocabulary tasks in testing without combining with VLMs or inserting new modules. CLIP-ZSS consists of two key modules: Global Learning Module (GLM) and Pixel Learning Module (PLM). GLM is proposed to probe the knowledge from the CLIP visual encoder by pulling the CLS token and the dense features from the image encoder of the same image and pushing others apart. Moreover, to enhance the ability to discriminate unseen categories, PLM consisting of pseudo labels and weight generation is designed. To generate semantically discriminated pseudo labels, a multi-scale K-Means with mask fusion working on the dense tokens is proposed. In pseudo weight generation, a synthesizer generating pseudo semantic features for the unannotated area is introduced. Experiments on three benchmarks show large performance gains compared with SOTA methods.

Motivation & Objective

• To address the limitations of existing CLIP-based zero-shot semantic segmentation methods that are constrained by fixed backbones or require explicit mask proposers.
• To improve performance by fully utilizing both annotated (seen) and unannotated (ignored) regions in images, which are typically discarded.
• To enable any segmentation model to perform zero-shot segmentation without architectural changes to CLIP or adding extra components like mask heads.
• To develop a flexible, plug-and-play framework that enhances zero-shot generalization by leveraging CLIP’s vision-language alignment.

Proposed method

• Introduces a Global Learning Module (GLM) that uses self-attention to align the CLIP CLS token with dense features from a segmentation model, enabling global knowledge distillation.
• Proposes a Pixel Learning Module (PLM) that applies multi-scale K-Means clustering on CLIP’s dense visual tokens to discover semantically coherent regions in unannotated areas.
• Uses a mask fusion algorithm to combine detected potential objects from ignored regions with ground-truth seen categories to form pseudo-labels.
• Employs a Vision-Language Adapter (VLA) with a transformer decoder to generate classifier weights for the newly discovered pseudo-objects.
• Maintains the original CLIP model unchanged and integrates only two lightweight modules, making it compatible with any pixel-wise segmentation backbone.
• Avoids explicit mask proposal networks and does not require fine-tuning of CLIP’s vision encoder.

Experimental results

Research questions

• RQ1 Can CLIP be effectively used as a knowledge distillation teacher for zero-shot semantic segmentation without modifying its architecture?
• RQ2 How can unannotated (ignored) regions in images be leveraged to improve zero-shot generalization in semantic segmentation?
• RQ3 Can dense CLIP tokens be used to discover and generate pseudo-annotations for unseen categories in unannotated regions?
• RQ4 Does a self-attention-based alignment between CLS tokens and dense features improve zero-shot recognition of unseen categories?
• RQ5 Can a transformer-based Vision-Language Adapter generate reliable classifier weights for pseudo-objects discovered in ignored regions?

Key findings

• CLIPTeacher achieves a 2.2% mIoU improvement on PASCAL VOC 2012 compared to state-of-the-art methods.
• The method improves mIoU by 1.3% on COCO-Stuff 164k, demonstrating strong generalization on large-scale datasets.
• On PASCAL Context, CLIPTeacher achieves a significant 8.8% mIoU gain, highlighting its effectiveness in complex scenes with many categories.
• Ablation studies show that removing the GLM leads to a 32.2% drop in mIoU (from 45.4% to 13.2%), proving its critical role in recognizing unseen categories.
• The GLM with self-attention outperforms max and average pooling by 9.9% and 3.4% respectively, confirming the importance of full feature utilization.
• The PLM with multi-scale K-Means and mask fusion outperforms MLP and raw feature baselines by 0.2% and 0.3% in mIoU, validating its design choice.