[Paper Review] CoBIT: A Contrastive Bi-directional Image-Text Generation Model

Haoxuan You, Mandy Guo+4 more|arXiv (Cornell University)|Mar 23, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

CoBIT proposes a unified vision-language foundation model that jointly pre-trains on contrastive learning, image-to-text generation, and text-to-image generation using a shared unicoder-decoder architecture. By enabling image and text unicoders to switch between encoding and decoding modes, CoBIT achieves state-of-the-art zero-shot performance, including 82.7% top-1 accuracy on ImageNet, 9.37 FID in text-to-image generation, and 44.8 CIDEr in image captioning.

ABSTRACT

The field of vision and language has witnessed a proliferation of pre-trained foundation models. Most existing methods are independently pre-trained with contrastive objective like CLIP, image-to-text generative objective like PaLI, or text-to-image generative objective like Parti. However, the three objectives can be pre-trained on the same data, image-text pairs, and intuitively they complement each other as contrasting provides global alignment capacity and generation grants fine-grained understanding. In this work, we present a Contrastive Bi-directional Image-Text generation model (CoBIT), which attempts to unify the three pre-training objectives in one framework. Specifically, CoBIT employs a novel unicoder-decoder structure, consisting of an image unicoder, a text unicoder and a cross-modal decoder. The image/text unicoders can switch between encoding and decoding in different tasks, enabling flexibility and shared knowledge that benefits both image-to-text and text-to-image generations. CoBIT achieves superior performance in image understanding, image-text understanding (Retrieval, Captioning, VQA, SNLI-VE) and text-based content creation, particularly in zero-shot scenarios. For instance, 82.7% in zero-shot ImageNet classification, 9.37 FID score in zero-shot text-to-image generation and 44.8 CIDEr in zero-shot captioning.

Motivation & Objective

• To unify three major pre-training objectives—contrastive learning, image-to-text generation, and text-to-image generation—into a single, coherent framework.
• To address the limitations of existing models that pre-train on only one or two objectives, despite sharing the same image-text pair data.
• To improve zero-shot and fine-tuned performance across vision and language tasks, including image understanding, retrieval, VQA, and content creation.
• To enable parameter-efficient knowledge sharing between unimodal encoding and decoding via a shared unicoder structure.
• To demonstrate that combining all three objectives in one model yields synergistic improvements without severe conflict between objectives.

Proposed method

• CoBIT employs a novel unicoder-decoder architecture with a shared image unicoder and text unicoder that can switch between encoding and decoding modes using shared parameters.
• The model uses a cross-attention decoder to fuse features from encoded image and text representations during generation tasks.
• During contrastive pre-training, both unicoders act as encoders to align image and text embeddings via contrastive loss.
• During image-to-text generation, the image unicoder encodes the image, and the text unicoder autoregressively decodes the caption, with cross-attention attending to the encoded image.
• During text-to-image generation, the text unicoder encodes the prompt, and the image unicoder autoregressively decodes image tokens, with cross-attention attending to the encoded text.
• The model is pre-trained on large-scale noisy web-crawled image-text data and image annotation data, with joint optimization of contrastive, I2T, and T2I losses.

Experimental results

Research questions

• RQ1 Can a single model effectively unify contrastive learning, image-to-text generation, and text-to-image generation using shared parameters?
• RQ2 How does joint pre-training of all three objectives affect performance across zero-shot and fine-tuned vision-language tasks?
• RQ3 To what extent do the image-to-text and text-to-image generation objectives conflict, and can they be harmonized in a unified framework?
• RQ4 Does the proposed unicoder mechanism, which shares parameters between encoding and decoding, improve performance and parameter efficiency compared to separate encoders?
• RQ5 Can CoBIT achieve strong zero-shot capabilities without fine-tuning, particularly in image classification, captioning, and text-to-image generation?

Key findings

• CoBIT achieves 82.7% zero-shot top-1 accuracy on ImageNet classification, demonstrating strong visual representation learning.
• In zero-shot text-to-image generation, CoBIT achieves a FID score of 9.37, indicating high-quality, diverse image generation from prompts.
• For zero-shot image captioning, CoBIT attains a CIDEr score of 44.8, indicating strong alignment between generated captions and reference descriptions.
• After fine-tuning, CoBIT achieves 86.44% linear probing accuracy on ImageNet, 4.62 FID on text-to-image generation, and 78.3 VQA score, showing strong transferability.
• Ablation studies show that while T2I and I2T objectives slightly conflict (e.g., VQA drops by 2.6 when T2I loss is added), the overall framework remains stable and effective.
• Training CoBIT from scratch without pre-initialization yields only a 0.3% drop in zero-shot ImageNet accuracy and 0.2 FID increase, confirming the model's trainability from scratch.