[Paper Review] SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training?

Hasan Abed Al Kader Hammoud, Hani Itani+4 more|arXiv (Cornell University)|Feb 2, 2024

Myasthenia Gravis and ThymomaMedicine3 citations

TL;DR

SynthCLIP proposes a fully synthetic training framework for CLIP models using large language models (LLMs) and text-to-image (TTI) diffusion models to generate scalable, high-quality, and safe image-caption pairs without real data. It achieves performance on par with real-data-trained CLIP models, demonstrating that fully synthetic data can enable scalable, controllable, and safe pretraining for vision-language models.

ABSTRACT

We present SynthCLIP, a CLIP model trained on entirely synthetic text-image pairs. Leveraging recent text-to-image (TTI) networks and large language models (LLM), we generate synthetic datasets of images and corresponding captions at scale, with no human intervention. In this work, we provide an analysis on CLIP models trained on synthetic data. We provide insights on the data generation strategy, number of samples required, scaling trends, and resulting properties. We also introduce SynthCI-30M, a purely synthetic dataset comprising 30 million captioned images. Our code, trained models, and data, are released as open source at https://github.com/hammoudhasan/SynthCLIP

Motivation & Objective

• To address the limitations of web-sourced text-image datasets, including data noise, long-tail distributions, and safety risks.
• To investigate whether CLIP models can be trained entirely on synthetic data without relying on any real data.
• To develop a scalable, automated pipeline for generating diverse, balanced, and safe synthetic text-image pairs at scale.
• To evaluate the impact of concept distribution and language model choice on downstream performance of synthetically trained CLIP models.
• To release a large-scale synthetic dataset (SynthCI-30M) and training code to enable reproducibility and further research in synthetic data training.

Proposed method

• Leverage a large concept bank of 500,000 concepts to ensure diversity and coverage of visual and linguistic elements.
• Use a large language model (e.g., Mistral 7B) to generate descriptive captions from sampled concepts via instruction-following prompting.
• Employ a text-to-image diffusion model (e.g., Stable Diffusion v1.5) to generate images conditioned on the LLM-generated captions.
• Ensure alignment between text and image by generating pairs end-to-end, guaranteeing semantic consistency.
• Apply safety filtering via built-in safeguards in state-of-the-art LLMs and TTI models to avoid harmful content.
• Train CLIP models on the fully synthetic dataset using contrastive learning to learn joint visual-textual representations.

Figure 1 : Advantages of SynthCLIP. Collecting text-image pairs from the internet often presents challenges: captions may not accurately match the images, specific classes may have limited representation due to scarcity, and there is a risk of encountering harmful content. We propose SynthCLIP, an a

Experimental results

Research questions

• RQ1 Can CLIP models trained exclusively on fully synthetic text-image pairs match the performance of models trained on real-world datasets?
• RQ2 How does the distribution of concepts in the training data affect the downstream performance of synthetically trained CLIP models?
• RQ3 What is the impact of using different large language models for caption generation on the quality of synthetic data and model performance?
• RQ4 Does random sampling of concepts lead to performance degradation compared to balanced or concept-specific sampling?
• RQ5 Can synthetic data generation pipelines achieve scalability and safety without human intervention, while maintaining model generalization?

Key findings

• SynthCLIP trained on 30 million synthetic image-caption pairs achieves performance comparable to CLIP models pre-trained on real datasets, with only minor performance gaps on downstream tasks.
• Balanced sampling of concepts from the concept bank leads to a 2.7% improvement in both image and text retrieval compared to random sampling, highlighting the importance of class distribution.
• Using a Mistral 7B model for caption generation outperforms a 33B Vicuna model, likely due to better instruction-following capabilities on the captioning task.
• Training on CC3M-derived concepts (40k subset) yields a 3.9% improvement in text retrieval and 1.6% in linear probing over the full concept bank, suggesting distribution bias in CC3M toward downstream-relevant concepts.
• Randomly selected concept subsets lead to worse performance than the full concept bank, indicating that concept relevance and coverage significantly impact model generalization.
• The release of SynthCI-30M, a fully synthetic dataset of 30 million paired images and captions, enables large-scale, safe, and scalable pretraining without real data.

Figure 2 : Pipeline Overview. From a set of concepts $\mathcal{C}$ (left), we obtain a set of synthetic captions $\mathcal{T}$ with an LLM, further refined to $\mathcal{T}^{*}$ by a filtering operation which subsamples $\mathcal{T}$ using balanced sampling (top). The generated captions are then used