[Paper Review] CoCa: Contrastive Captioners are Image-Text Foundation Models

Jiahui Yu, Zirui Wang+4 more|arXiv (Cornell University)|May 4, 2022

Multimodal Machine Learning ApplicationsComputer Science515 citations

TL;DR

CoCa unifies contrastive and generative vision-language pretraining in a single encoder-decoder model, enabling strong zero-shot and transfer performance across visual recognition, cross-modal retrieval, and multimodal understanding with a single pretrained checkpoint.

ABSTRACT

Exploring large-scale pretrained foundation models is of significant interest in computer vision because these models can be quickly transferred to many downstream tasks. This paper presents Contrastive Captioner (CoCa), a minimalist design to pretrain an image-text encoder-decoder foundation model jointly with contrastive loss and captioning loss, thereby subsuming model capabilities from contrastive approaches like CLIP and generative methods like SimVLM. In contrast to standard encoder-decoder transformers where all decoder layers attend to encoder outputs, CoCa omits cross-attention in the first half of decoder layers to encode unimodal text representations, and cascades the remaining decoder layers which cross-attend to the image encoder for multimodal image-text representations. We apply a contrastive loss between unimodal image and text embeddings, in addition to a captioning loss on the multimodal decoder outputs which predicts text tokens autoregressively. By sharing the same computational graph, the two training objectives are computed efficiently with minimal overhead. CoCa is pretrained end-to-end and from scratch on both web-scale alt-text data and annotated images by treating all labels simply as text, seamlessly unifying natural language supervision for representation learning. Empirically, CoCa achieves state-of-the-art performance with zero-shot transfer or minimal task-specific adaptation on a broad range of downstream tasks, spanning visual recognition (ImageNet, Kinetics-400/600/700, Moments-in-Time), crossmodal retrieval (MSCOCO, Flickr30K, MSR-VTT), multimodal understanding (VQA, SNLI-VE, NLVR2), and image captioning (MSCOCO, NoCaps). Notably on ImageNet classification, CoCa obtains 86.3% zero-shot top-1 accuracy, 90.6% with a frozen encoder and learned classification head, and new state-of-the-art 91.0% top-1 accuracy on ImageNet with a finetuned encoder.

Motivation & Objective

• Motivate and develop a single image-text foundation model that subsumes single-encoder, dual-encoder, and encoder-decoder paradigms.
• Introduce a decoupled decoder architecture that yields unimodal text representations for contrastive learning and multimodal representations for captioning.
• Train jointly with both contrastive and captioning losses in a single, end-to-end pretraining stage from scratch on web-scale alt-text and annotated images.
• Show that the unified model achieves state-of-the-art or competitive performance on a wide set of downstream tasks with zero-shot transfer or minimal task-specific adaptation.

Proposed method

• Propose Contrastive Captioners (CoCa) with a two-part decoder: unimodal text decoder (no cross-attention) and multimodal decoder (cross-attends to image encoder).
• Train with a joint objective L_CoCa = lambda_Con * L_Con + lambda_Cap * L_Cap, combining a contrastive loss between unimodal image and text embeddings and a captioning loss on multimodal decoder outputs.
• Use a decoupled training graph allowing shared forward pass for both objectives; leverage a single CLS-embedded image token for contrastive learning and an autoregressive captioning stream for text generation.
• Pretrain from scratch on web-scale data (JFT-3B labels as text + ALIGN alt-text) with a 65,536 pair batch size and a unified pretraining regime.
• Adopt task-specific attentional pooling to produce both global (contrastive) and region-level (captioning/multimodal) visual representations.
• Provide model variants (CoCa-Base, CoCa-Large, CoCa) with detailed encoder/decoder specs and parameter counts; evaluate zero-shot, frozen-encoder, and finetuned settings.

Experimental results

Research questions

• RQ1 Can a single image-text foundation model trained with both contrastive and captioning objectives achieve state-of-the-art performance across visual recognition, cross-modal retrieval, and multimodal understanding tasks?
• RQ2 Does decoupling the decoder into unimodal and multimodal components enable efficient joint training and versatile downstream transfer without multi-stage pretraining?
• RQ3 How do zero-shot transfer, frozen-feature evaluation, and finetuning compare for a unified CoCa model across diverse benchmarks?
• RQ4 What are the effects of task-specific attentional pooling on pooling visual features for different downstream tasks?
• RQ5 Can training on both annotated images and noisy image-text data yield robust crossmodal alignment and generation capabilities without task-specific pretraining stages?

Key findings

• CoCa achieves state-of-the-art or competitive results across a broad set of tasks, including visual recognition, cross-modal retrieval, multimodal understanding, and image captioning.
• On ImageNet, CoCa achieves 86.3% zero-shot top-1 accuracy, 90.6% with a frozen encoder, and 91.0% with finetuning.
• In zero-shot cross-modal retrieval, CoCa outperforms prior methods on MSCOCO and Flickr30K, with strong retrieval metrics.
• For video retrieval and action recognition, CoCa demonstrates competitive or superior performance in zero-shot and frozen-encoder settings, and improves with finetuning.
• On multimodal understanding benchmarks (VQA, SNLI-VE, NLVR2) CoCa outperforms strong vision-language baselines.
• In image captioning (MSCOCO, NoCaps), CoCa achieves strong BLEU/METEOR/CIDEr/SPICE scores without CIDEr-specific optimization, and NoCaps results set new state-of-the-art for generated captions.