[Paper Review] Contrastive Language-Image Pre-training for the Italian Language

Federico Bianchi, Giuseppe Attanasio+4 more|arXiv (Cornell University)|Aug 19, 2021

Linguistic Studies and Language Acquisition15 references21 citations

TL;DR

This paper presents CLIP-Italian, the first contrastive language-image pre-training model fine-tuned specifically for the Italian language using 1.4 million image-text pairs. Trained by adapting pre-existing vision and text encoders with language-specific fine-tuning, CLIP-Italian outperforms the multilingual CLIP model on zero-shot image classification and image retrieval tasks, demonstrating superior performance despite limited data and computational resources.

ABSTRACT

CLIP (Contrastive Language-Image Pre-training) is a very recent multi-modal model that jointly learns representations of images and texts. The model is trained on a massive amount of English data and shows impressive performance on zero-shot classification tasks. Training the same model on a different language is not trivial, since data in other languages might be not enough and the model needs high-quality translations of the texts to guarantee a good performance. In this paper, we present the first CLIP model for the Italian Language (CLIP-Italian), trained on more than 1.4 million image-text pairs. Results show that CLIP-Italian outperforms the multilingual CLIP model on the tasks of image retrieval and zero-shot classification.

Motivation & Objective

• To develop a high-performing, language-specific contrastive language-image pre-training model for the Italian language.
• To address the performance gap between monolingual and multilingual CLIP models by creating a dedicated Italian variant.
• To build and release the largest publicly available multi-modal dataset for Italian, comprising 1.4 million image-caption pairs.
• To demonstrate that a language-specific CLIP model can outperform its multilingual counterpart on downstream tasks.
• To provide a transparent, accessible, and reproducible model and demo for the research community.

Proposed method

• Fine-tuning a pre-trained CLIP architecture by adapting vision and text encoders using a curated, multilingual dataset of 1.4 million Italian image-text pairs.
• Combining data from four sources: WIT (Wikipedia-based), MSCOCO-IT (translated MSCOCO), Conceptual Captions, and a new Italian-specific web-crawled dataset.
• Applying prompt engineering with a standardized template: 'A photo of {caption}' to ensure consistent textual input for zero-shot classification.
• Training the model using a contrastive loss that aligns image and text embeddings in a shared 512-dimensional space.
• Leveraging Hugging Face and Google Cloud TPUs for training, with model weights and inference demo released publicly.
• Evaluating performance on zero-shot image classification (ImageNet-1000) and image retrieval (Unsplash25K).

Experimental results

Research questions

• RQ1 Can a fine-tuned, language-specific CLIP model outperform a multilingual CLIP model on Italian zero-shot image classification and retrieval tasks?
• RQ2 How does the performance of a CLIP model trained on 1.4 million Italian image-text pairs compare to the original English CLIP and multilingual CLIP?
• RQ3 What are the limitations and biases of a fine-tuned CLIP model in understanding complex or ambiguous visual descriptions in Italian?
• RQ4 To what extent does prompt engineering influence zero-shot classification accuracy in a low-resource language like Italian?
• RQ5 Can a model trained on limited, non-English data achieve competitive performance compared to models trained on massive English datasets?

Key findings

• CLIP-Italian achieved an accuracy of 22.11% on top-1 zero-shot image classification on ImageNet-1000, outperforming the multilingual CLIP (20.15%).
• On the same benchmark, CLIP-Italian reached 43.69% accuracy on top-5, compared to 36.75% for mCLIP.
• For image retrieval on the Unsplash25K dataset, CLIP-Italian correctly retrieved relevant images for queries like 'due cani sulla neve' (two dogs on the snow), demonstrating effective semantic and visual grounding.
• The model showed moderate numerical reasoning, correctly identifying up to three distinct or repeated objects in a scene, but performance dropped significantly with more than three elements.
• Qualitative analysis revealed model biases, such as associating 'un topolino' (a small mouse) with a small hedgehog, indicating learned stereotypes or data biases.
• Despite training on only 1.4 million pairs (vs. 400 million for the original CLIP), CLIP-Italian achieved state-of-the-art performance for Italian, suggesting strong efficiency and effectiveness of language-specific fine-tuning.