[Paper Review] GIT: A Generative Image-to-text Transformer for Vision and Language

Jianfeng Wang, Zhengyuan Yang+7 more|arXiv (Cornell University)|May 27, 2022

Multimodal Machine Learning ApplicationsComputer Science209 citations

TL;DR

GIT is a single-encoder image and single-decoder text generator that pre-trains on 0.8B image-text pairs to unify image/video captioning and VQA, achieving new state-of-the-art results without object detectors or OCR. It also demonstrates generation-based image classification and strong scene-text capabilities.

ABSTRACT

In this paper, we design and train a Generative Image-to-text Transformer, GIT, to unify vision-language tasks such as image/video captioning and question answering. While generative models provide a consistent network architecture between pre-training and fine-tuning, existing work typically contains complex structures (uni/multi-modal encoder/decoder) and depends on external modules such as object detectors/taggers and optical character recognition (OCR). In GIT, we simplify the architecture as one image encoder and one text decoder under a single language modeling task. We also scale up the pre-training data and the model size to boost the model performance. Without bells and whistles, our GIT establishes new state of the arts on 12 challenging benchmarks with a large margin. For instance, our model surpasses the human performance for the first time on TextCaps (138.2 vs. 125.5 in CIDEr). Furthermore, we present a new scheme of generation-based image classification and scene text recognition, achieving decent performance on standard benchmarks. Codes are released at \url{https://github.com/microsoft/GenerativeImage2Text}.

Motivation & Objective

• Demonstrate a simple, unified Generative Image-to-text Transformer (GIT) architecture for vision-language tasks.
• Show that pre-training on large-scale image-text data without detectors/OCR yields state-of-the-art results on captioning, QA, and video tasks.
• Explore generation-based image classification and scene text recognition using the same model.
• Investigate the impact of model size and data scale on VL task performance.
• Provide insights into training dynamics when using a decoder that is randomly initialized and trained end-to-end.

Proposed method

• Use one Swin-like vision encoder pre-trained on large image-text pairs as the image encoder.
• Use a transformer text decoder trained with a language modeling objective to generate descriptions from image features.
• Pre-train on 0.8B image-text pairs with LM loss (no MLM), using 2 epochs.
• Fine-tune for captioning, VQA, and video tasks by treating targets as generated text (question as prompt for VQA).
• Handle video by sampling multiple frames, encoding independently, and concatenating with a learnable temporal embedding.
• Evaluate generation-based image classification by predicting class names as captions without a fixed vocabulary.

Experimental results

Research questions

• RQ1 Can a minimal architecture consisting of one image encoder and one text decoder achieve state-of-the-art results across VL tasks without relying on detectors, tags, or OCR?
• RQ2 How does scaling pre-training data and model size affect VL task performance across image, video, and QA benchmarks?
• RQ3 Is generation-based learning capable of achieving competitive or state-of-the-art results on scene-text understanding and image classification without predefined vocabularies?
• RQ4 To what extent can a single generative model transfer from image captioning to video captioning and VQA with minimal architectural changes?

Key findings

• GIT achieves new state-of-the-art across multiple image/video captioning and QA benchmarks.
• On TextCaps, GIT surpasses human performance for the first time (CIDEr: 138.2 vs 125.5).
• GIT surpasses prior SOTA on several benchmarks with a 0.7B parameter model without external detectors or OCR.
• Video tasks benefit from the same architecture, yielding strong results without dedicated video encoders.
• A generation-based ImageNet classification scheme using caption-like outputs achieves competitive top-1 accuracy (88.79%).
• Scaling pre-training data and model size yields substantial gains across many tasks.