[Paper Review] CLIP4IDC: CLIP for Image Difference Captioning

Zixin Guo, Tzu-Jui Julius Wang+1 more|arXiv (Cornell University)|Jun 1, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

CLIP4IDC proposes a two-stage adaptation-and-fine-tune strategy to transfer CLIP for Image Difference Captioning (IDC), improving visual representation of image pair differences through contrastive retrieval loss before fine-tuning a captioning head. It achieves state-of-the-art performance on three IDC benchmarks by narrowing the domain and objective gap between CLIP pre-training and IDC fine-tuning.

ABSTRACT

Image Difference Captioning (IDC) aims at generating sentences to describe differences between two similar-looking images. Conventional approaches learn an IDC model with a pre-trained and usually frozen visual feature extractor. Accordingly, two major issues may arise: (1) a large domain gap usually exists between the pre-training datasets used for training such a visual encoder and that of the downstream IDC task, and (2) the visual feature extractor, when separately encoding two images, often does not effectively encode the visual changes between two images. Due to the excellent zero-shot performance of the recently proposed CLIP, we thus propose CLIP4IDC to transfer a CLIP model for the IDC task to address those issues. Different from directly fine-tuning CLIP to generate sentences, we introduce an adaptation training process to adapt CLIP's visual encoder to capture and align differences in image pairs based on the textual descriptions. Experiments on three IDC benchmark datasets, CLEVR-Change, Spot-the-Diff, and Image-Editing-Request, demonstrate the effectiveness of CLIP4IDC.

Motivation & Objective

• Address the domain gap between CLIP pre-training and downstream IDC tasks, where pre-trained visual encoders often fail to capture fine-grained differences.
• Overcome the limitation of frozen visual feature extractors that do not model inter-image changes effectively.
• Improve alignment between visual representations of image pairs and textual descriptions of differences.
• Develop a method that leverages CLIP’s zero-shot capabilities while adapting it specifically for IDC through targeted adaptation.
• Demonstrate that retrieval-based adaptation effectively improves downstream captioning performance on diverse IDC datasets.

Proposed method

• Propose a two-stage training pipeline: (1) adaptation with contrastive retrieval loss to align image pair representations with textual descriptions of differences, and (2) fine-tuning a captioning head on the adapted visual encoder.
• Introduce two modules: an intra-encoder to extract and compare differences within a pair of images, and an inter-encoder to align the pair with corresponding textual descriptions.
• Use a contrastive learning objective where positive pairs (image pair ↔ description) are pulled together, and negative pairs are pushed apart.
• Train the adaptation stage using image-pair-to-text and text-to-image-pair retrieval tasks to ensure fine-grained difference modeling.
• Fine-tune the adapted CLIP visual encoder with a Transformer-based captioning head trained from scratch on raw image pixels.
• Optimize the model end-to-end in the adaptation stage and then jointly fine-tune the visual encoder and captioning head.

Experimental results

Research questions

• RQ1 Can CLIP be effectively adapted for Image Difference Captioning through a retrieval-based adaptation stage that captures fine-grained visual changes?
• RQ2 Does the proposed adaptation-and-fine-tune strategy narrow the objective and domain gap between CLIP pre-training and IDC better than direct fine-tuning or feature-based fine-tuning?
• RQ3 How does the number of layers in the intra and inter modules affect retrieval and captioning performance?
• RQ4 To what extent does retrieval performance on image-pair-to-text and text-to-image-pair tasks correlate with downstream captioning accuracy?
• RQ5 Can CLIP4IDC generalize across synthetic and real-world IDC datasets, including mixed real–synthetic benchmarks?

Key findings

• CLIP4IDC achieves SOTA performance on all three IDC benchmarks: CLEVR-Change, Spot-the-Diff, and Image-Editing-Request.
• On CLEVR-Change, CLIP4IDC achieves 46.4% R@1 and 86.6% R@10 in image-pair-to-text retrieval, indicating strong alignment of visual differences with descriptions.
• The model achieves 37.4 BERTScore, 146.5 ROUGE-L, and 75.2 CIDEr on the CLEVR-Change test split, outperforming strong baselines.
• Retrieval performance on adaptation tasks strongly correlates with captioning performance, validating the use of retrieval metrics as a proxy for IDC quality.
• Allocating more layers to the intra-encoder module improves performance, and removing inter-encoder layers significantly degrades results, proving its necessity for modeling inter-image differences.
• The ablation study shows that removing inter-encoder layers reduces R@1 to 2.3% and increases MdR to 182.0, confirming that global inter-image alignment is critical for success.