[Paper Review] FILIP: Fine-grained Interactive Language-Image Pre-Training

Lewei Yao, Runhui Huang+8 more|arXiv (Cornell University)|Nov 9, 2021

Multimodal Machine Learning Applications29 references205 citations

TL;DR

FILIP introduces a fine-grained cross-modal late interaction using token-wise maximum similarities for image-text alignment, achieving strong zero-shot and retrieval performance with offline pre-computation and a new 300M-image-text dataset (FILIP300M).

ABSTRACT

Unsupervised large-scale vision-language pre-training has shown promising advances on various downstream tasks. Existing methods often model the cross-modal interaction either via the similarity of the global feature of each modality which misses sufficient information, or finer-grained interactions using cross/self-attention upon visual and textual tokens. However, cross/self-attention suffers from inferior efficiency in both training and inference. In this paper, we introduce a large-scale Fine-grained Interactive Language-Image Pre-training (FILIP) to achieve finer-level alignment through a cross-modal late interaction mechanism, which uses a token-wise maximum similarity between visual and textual tokens to guide the contrastive objective. FILIP successfully leverages the finer-grained expressiveness between image patches and textual words by modifying only contrastive loss, while simultaneously gaining the ability to pre-compute image and text representations offline at inference, keeping both large-scale training and inference efficient. Furthermore, we construct a new large-scale image-text pair dataset called FILIP300M for pre-training. Experiments show that FILIP achieves state-of-the-art performance on multiple downstream vision-language tasks including zero-shot image classification and image-text retrieval. The visualization on word-patch alignment further shows that FILIP can learn meaningful fine-grained features with promising localization ability.

Motivation & Objective

• Motivate fine-grained vision-language alignment beyond global feature similarity.
• Provide an efficient dual-stream VLP framework with cross-modal late interaction.
• Demonstrate that token-wise max similarity improves fine-grained alignment while enabling offline pre-computation.
• Show efficacy with a new large-scale dataset FILIP300M (300M image-text pairs).
• Explore data augmentation and prompt-template strategies to boost zero-shot and retrieval performance.

Proposed method

• Dual-stream Vision Transformer encoders for image and text.
• Cross-modal late interaction that computes token-wise maximum similarities between image patches and textual tokens, then averages over non-padded tokens to form image-text similarities (Equations 3–5).
• Contrastive loss over image-to-text and text-to-image pairs, with a 1/2 sum of the two directional losses (Equation 1).
• Use of prompt templates and prompt ensemble to improve zero-shot classification, with mean token-wise similarity for ensemble (Section 3.1.2).
• Image and text augmentation including AutoAugment and back-translation for text to create additional pairs (Section 3.2).
• Pre-training on FILIP300M (~340M cleaned pairs) with dataset filtering and integration of additional public datasets (CC3M, CC12M, YFCC100M) (Section 3.3).

Experimental results

Research questions

• RQ1 Can a fine-grained token-wise cross-modal interaction improve vision-language alignment without sacrificing the efficiency of dual-stream models?
• RQ2 Does replacing global feature similarity with token-level max similarity enhance zero-shot and retrieval performance across standard datasets?
• RQ3 How do data augmentation and prompt templates affect zero-shot classification and retrieval in large-scale VLP?
• RQ4 What is the impact of using a large yet efficiently-leveraged dataset (FILIP300M) on downstream vision-language tasks?
• RQ5 Is offline pre-computation of image/text representations compatible with fine-grained interactions for scalable inference?

Key findings

• FILIP achieves state-of-the-art results on multiple downstream tasks including zero-shot image classification and image-text retrieval.
• On zero-shot 12-dataset ImageNet classification, FILIP base improves average top-1 by 5.6 points and FILIP large by 3.0 points over CLIP with less training data (340M vs 400M).
• In zero-shot image-text retrieval, FILIP outperforms CLIP on MSCOCO with notable gains (e.g., image-to-text R@1: 60.1 vs 59.? in CLIP baseline; exact numbers in table).
• FILIP attains competitive or superior results across Flickr30K and MSCOCO in both zero-shot and fine-tuned image-text retrieval settings, achieving state-of-the-art on several metrics (Tables 2–3).
• Word-patch alignment visualizations show meaningful fine-grained alignment, e.g., correctly localizing components like “electric” and “locomotive” in objects.
• Ablation studies confirm that image/text augmentation and cross-modal late interaction each contribute to performance gains (Table 4).
• Efficiency analyses indicate that with optimizations (reduced embedding size, fp16 precision, token-selection), late interaction remains practical with modest memory/time costs (Table 5).