[Paper Review] S-CLIP: Semi-supervised Vision-Language Learning using Few Specialist Captions

Sangwoo Mo, Minkyu Kim+2 more|arXiv (Cornell University)|May 23, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

S-CLIP proposes a semi-supervised vision-language pre-training method that leverages few paired image-text captions and abundant unpaired images to improve CLIP in specialized domains. It introduces caption-level pseudo-labeling via optimal transport and keyword-level pseudo-labeling through partial label learning, achieving 10% higher zero-shot accuracy and 4% better retrieval on remote sensing benchmarks using only one-third the labeled data compared to supervised fine-tuning.

ABSTRACT

Vision-language models, such as contrastive language-image pre-training (CLIP), have demonstrated impressive results in natural image domains. However, these models often struggle when applied to specialized domains like remote sensing, and adapting to such domains is challenging due to the limited number of image-text pairs available for training. To address this, we propose S-CLIP, a semi-supervised learning method for training CLIP that utilizes additional unpaired images. S-CLIP employs two pseudo-labeling strategies specifically designed for contrastive learning and the language modality. The caption-level pseudo-label is given by a combination of captions of paired images, obtained by solving an optimal transport problem between unpaired and paired images. The keyword-level pseudo-label is given by a keyword in the caption of the nearest paired image, trained through partial label learning that assumes a candidate set of labels for supervision instead of the exact one. By combining these objectives, S-CLIP significantly enhances the training of CLIP using only a few image-text pairs, as demonstrated in various specialist domains, including remote sensing, fashion, scientific figures, and comics. For instance, S-CLIP improves CLIP by 10% for zero-shot classification and 4% for image-text retrieval on the remote sensing benchmark, matching the performance of supervised CLIP while using three times fewer image-text pairs.

Motivation & Objective

• To improve vision-language pre-training in specialized domains like remote sensing, where large-scale paired data is scarce.
• To address the limitations of naive pseudo-labeling in vision-language models, where captions vary significantly even for similar images.
• To develop a semi-supervised learning framework that effectively utilizes unpaired images alongside only a few paired image-text examples.
• To design pseudo-labeling strategies tailored for contrastive learning and language modality, ensuring robustness to distribution shifts and caption diversity.
• To achieve performance comparable to fully supervised CLIP while using significantly fewer labeled image-text pairs.

Proposed method

• Caption-level pseudo-labeling is generated by solving an optimal transport problem between unpaired and paired images, producing a probability distribution over labeled captions to guide global semantic alignment.
• Keyword-level pseudo-labeling constructs a candidate set of keywords from the nearest labeled image, treating the task as partial label learning to model local visual-semantic components.
• The method combines both pseudo-labeling objectives: caption-level for image-text retrieval and keyword-level for zero-shot classification, enabling complementary learning.
• The model is trained using contrastive learning with both supervised and pseudo-labeled data, where pseudo-labels are refined iteratively during training.
• Batch sampling is optimized by sorting mini-batches based on visual similarity to improve pseudo-label quality and training stability.
• The approach avoids reliance on external detectors or class annotations, making it applicable to domains with limited annotation infrastructure.

Experimental results

Research questions

• RQ1 Can semi-supervised learning with few paired captions and abundant unpaired images significantly improve vision-language models in specialized domains?
• RQ2 How can pseudo-labeling be designed to handle the high diversity and uniqueness of image captions, avoiding collapse to nearest-caption assignment?
• RQ3 To what extent can caption-level and keyword-level pseudo-labeling jointly enhance zero-shot classification and image-text retrieval performance?
• RQ4 Does optimal transport-based caption pseudo-labeling outperform naive nearest-caption assignment in the presence of distribution shifts between labeled and unlabeled data?
• RQ5 Can partial label learning for keywords improve model generalization when exact label supervision is unavailable?

Key findings

• S-CLIP improves zero-shot classification accuracy by 10% and image-text retrieval by 4% on the RSICD remote sensing benchmark compared to standard CLIP fine-tuning.
• S-CLIP matches the performance of fully supervised CLIP while using only one-third of the image-text pairs, demonstrating strong data efficiency.
• The ablation study confirms that both caption-level and keyword-level pseudo-labeling contribute significantly, with the combination yielding the best results.
• Sorted mini-batch sampling improves performance by 1.5% in zero-shot accuracy and 0.6% in retrieval, showing the benefit of informative batch composition.
• Baseline ablation shows that keyword information and semi-supervised learning both contribute independently, with combined use yielding 70.6% zero-shot accuracy and 10.1% retrieval.
• The method remains effective even when class names are absent, as 94.8% of captions contain more than two keywords, validating the need for partial label learning.