[Paper Review] Interpreting CLIP's Image Representation via Text-Based Decomposition

Yossi Gandelsman, Alexei A. Efros+1 more|arXiv (Cornell University)|Oct 9, 2023

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes a text-based decomposition method to interpret CLIP's image representation by attributing contributions to individual attention heads and image patches. By leveraging CLIP's joint vision-language space, it identifies interpretable, property-specific roles for attention heads (e.g., shape, location) and enables zero-shot image segmentation with 54.50 mIoU, outperforming prior methods.

ABSTRACT

We investigate the CLIP image encoder by analyzing how individual model components affect the final representation. We decompose the image representation as a sum across individual image patches, model layers, and attention heads, and use CLIP's text representation to interpret the summands. Interpreting the attention heads, we characterize each head's role by automatically finding text representations that span its output space, which reveals property-specific roles for many heads (e.g. location or shape). Next, interpreting the image patches, we uncover an emergent spatial localization within CLIP. Finally, we use this understanding to remove spurious features from CLIP and to create a strong zero-shot image segmenter. Our results indicate that a scalable understanding of transformer models is attainable and can be used to repair and improve models.

Motivation & Objective

• To understand how individual components of CLIP's vision transformer contribute to the final image representation.
• To interpret attention heads and image patches using natural language descriptions from CLIP's text encoder.
• To reduce spurious correlations in downstream tasks by ablating heads associated with spurious cues.
• To develop a strong zero-shot image segmentation method based on interpretable, property-specific representations.
• To demonstrate that scalable interpretation of transformer models can lead to model repair and performance improvement.

Proposed method

• Decomposes CLIP's image representation as a sum across image patches, attention heads, and model layers using residual connections.
• Applies the TextSpan algorithm to find a text-interpretable basis for each attention head’s output space, labeling directions with natural language descriptions.
• Uses spatial attention mechanisms to attribute contributions to specific image locations for each basis direction.
• Jointly decomposes representation across heads and image tokens to visualize which regions contribute to each text-labeled direction.
• Employs ablation of specific attention heads to remove spurious features, validated on the Waterbirds dataset.
• Constructs a zero-shot image segmenter by combining spatial decomposition with property-specific head representations.

Experimental results

Research questions

• RQ1 Which attention heads in CLIP-ViT are responsible for encoding specific visual attributes such as shape, color, or location?
• RQ2 Can we automatically discover natural language descriptions that span the output space of individual attention heads?
• RQ3 Does emergent spatial localization arise in CLIP's representation, and can it be leveraged for zero-shot segmentation?
• RQ4 Can removing heads associated with spurious cues improve worst-group accuracy in biased datasets?
• RQ5 Can joint decomposition of heads and image tokens validate the interpretability of discovered text-based directions?

Key findings

• Ablating all but the last four attention layers has minimal impact on zero-shot classification accuracy, indicating that late layers primarily construct the image representation.
• TextSpan identifies interpretable, property-specific roles for attention heads—e.g., one head specializes in shapes like 'isosceles triangle' or 'oval'.
• Removing heads associated with spurious cues improves worst-group accuracy on the Waterbirds dataset from 48% to 73%.
• The joint decomposition of heads and image tokens reveals that regions matching a text description (e.g., Eiffel Tower) are the primary contributors to the corresponding basis direction.
• The proposed zero-shot image segmenter achieves 54.50 mIoU on ImageNet-segmentation, outperforming existing methods including LRP, GradCAM, and rollout.
• The method enables retrieval based on discovered semantic similarities such as color, location, and texture, demonstrating utility beyond classification.