[Paper Review] OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework

Peng Wang, Yang An+8 more|arXiv (Cornell University)|Feb 7, 2022

Multimodal Machine Learning ApplicationsComputer Science258 citations

TL;DR

OFA presents a Task-Agnostic and Modality-Agnostic Seq2Seq framework that unifies architectures, tasks, and modalities for multimodal and unimodal pretraining, achieving SOTA on several vision-language tasks with published code.

ABSTRACT

In this work, we pursue a unified paradigm for multimodal pretraining to break the scaffolds of complex task/modality-specific customization. We propose OFA, a Task-Agnostic and Modality-Agnostic framework that supports Task Comprehensiveness. OFA unifies a diverse set of cross-modal and unimodal tasks, including image generation, visual grounding, image captioning, image classification, language modeling, etc., in a simple sequence-to-sequence learning framework. OFA follows the instruction-based learning in both pretraining and finetuning stages, requiring no extra task-specific layers for downstream tasks. In comparison with the recent state-of-the-art vision & language models that rely on extremely large cross-modal datasets, OFA is pretrained on only 20M publicly available image-text pairs. Despite its simplicity and relatively small-scale training data, OFA achieves new SOTAs in a series of cross-modal tasks while attaining highly competitive performances on uni-modal tasks. Our further analysis indicates that OFA can also effectively transfer to unseen tasks and unseen domains. Our code and models are publicly available at https://github.com/OFA-Sys/OFA.

Motivation & Objective

• Pursue a unified, task-agnostic and modality-agnostic multimodal pretraining paradigm.
• Eliminate task-specific heads and adapters to enable zero-shot and cross-domain transfer.
• Unify a broad spectrum of tasks (generation and understanding) across vision, language, and cross-modality in a single framework.
• Demonstrate competitive or state-of-the-art performance on cross-modal and unimodal benchmarks with relatively modest data.

Proposed method

• Represent diverse modalities in a shared, token-based vocabulary using image codes, region tokens, and BPE text tokens.
• Use a Transformer encoder-decoder as a single architecture for pretraining, finetuning, and inference across all tasks.
• Form all pretraining and downstream tasks as sequence-to-sequence generation with handcrafted instructions to provide task guidance.
• Pretrain on 20M public image-text pairs with multitask objectives including visual grounding, grounded captioning, image-text matching, image captioning, VQA, object detection, and image infilling, plus language text infilling for pure NLP tasks.
• Introduce a Trie-based decoding strategy to improve efficiency and accuracy for classification-like outputs.

Experimental results

Research questions

• RQ1 Can a single Seq2Seq model with a unified instruction representation handle both unimodal and multimodal tasks across vision and language?
• RQ2 Does removing task-specific heads/adapters and enforcing modality-agnostic representations enable effective zero-shot and cross-domain transfer?
• RQ3 How does multitask pretraining with diverse vision-language tasks impact downstream performance on VQA, captioning, grounding, and unimodal benchmarks?
• RQ4 What are the trade-offs of a smaller versus larger OFA model in terms of cross-modal and unimodal performance?
• RQ5 To what extent can OFA transfer to unseen tasks/domains without finetuning?

Key findings

• OFA achieves 82.0 on VQA test-std and 91.0/91.2 on SNLI-VE test sets, outperforming prior SOTAs on cross-modal understanding tasks.
• On MSCOCO image captioning (Karpathy split), OFA reaches CIDEr 154.9 with CIDEr optimization, surpassing prior SOTA methods like SimVLM Huge and LEMON.
• For referring expression comprehension, OFA attains strong gains: RefCOCO testA 90.67, RefCOCO+ testA 87.68, RefCOCOg test-u 88.78, surpassing prior SOTA by several points.
• In text-to-image generation, OFA achieves FID 10.5, CLIPSIM 34.4, and IS 31.1, outperforming CogView and NÜWA with a smaller sampling size.
• Unimodal tasks see competitive performance: GLUE (SST-2, RTE, MRPC, QQP, QNLI, MNLI) and Gigaword abstractive summarization achieve strong scores close to or surpassing modality-specific baselines, with OFA matching or exceeding many state-of-the-art models in natural language tasks and achieving ImageNet-1K finetuning accuracy of 85.6% for OFA Large.
• Zero-shot learning demonstrates competitive performance on GLUE tasks and SNLI-VE, with notable transfer to unseen tasks like grounded QA and VQA in out-of-domain images.