[Paper Review] Flamingo: a Visual Language Model for Few-Shot Learning

Jean-Baptiste Alayrac, Jeff Donahue+25 more|arXiv (Cornell University)|Apr 29, 2022

Multimodal Machine Learning Applications1,238 citations

TL;DR

Flamingo is a Visual Language Model that achieves strong few-shot learning across diverse image/video and language tasks by conditioning a frozen large language model on interleaved visual inputs via a Perceiver-based visual resampler and gated cross-attention, enabling open-ended generation without task-specific fine-tuning.

ABSTRACT

Building models that can be rapidly adapted to novel tasks using only a handful of annotated examples is an open challenge for multimodal machine learning research. We introduce Flamingo, a family of Visual Language Models (VLM) with this ability. We propose key architectural innovations to: (i) bridge powerful pretrained vision-only and language-only models, (ii) handle sequences of arbitrarily interleaved visual and textual data, and (iii) seamlessly ingest images or videos as inputs. Thanks to their flexibility, Flamingo models can be trained on large-scale multimodal web corpora containing arbitrarily interleaved text and images, which is key to endow them with in-context few-shot learning capabilities. We perform a thorough evaluation of our models, exploring and measuring their ability to rapidly adapt to a variety of image and video tasks. These include open-ended tasks such as visual question-answering, where the model is prompted with a question which it has to answer; captioning tasks, which evaluate the ability to describe a scene or an event; and close-ended tasks such as multiple-choice visual question-answering. For tasks lying anywhere on this spectrum, a single Flamingo model can achieve a new state of the art with few-shot learning, simply by prompting the model with task-specific examples. On numerous benchmarks, Flamingo outperforms models fine-tuned on thousands of times more task-specific data.

Motivation & Objective

• Motivate rapid adaptation to novel multimodal tasks with minimal annotated data.
• Bridge pretrained vision-only models and language models to handle interleaved visual and textual data.
• Enable open-ended language generation conditioned on images/videos rather than fixed outputs.
• Evaluate few-shot performance across diverse vision-language benchmarks and analyze design choices.

Proposed method

• Use a frozen large language model (Chinchilla) as the backbone and insert trainable cross-attention blocks to condition it on visual inputs.
• Represent images/videos with a Perceiver Resampler that produces a fixed number of visual tokens from variable-size feature maps.
• Interleave text and visual tokens in prompts so the model predicts the next text token conditioned on prior text and preceding visuals.
• Train on a mixture of web-scraped vision-language data (interleaved HTML text with images, image-text pairs, and video-text pairs) to support in-context learning.
• Adopt a tanh-gated cross-attention mechanism to fuse visual information while preserving LM weights and stability.

Experimental results

Research questions

• RQ1 Can a vision-language model perform diverse multimodal tasks in a few-shot setting without task-specific fine-tuning?
• RQ2 What architectural components best enable conditioning a frozen LM on interleaved visual inputs (images/videos) from variable input lengths?
• RQ3 How does training on a mixture of interleaved and paired vision-language data affect generalization and few-shot adaptation?
• RQ4 To what extent can in-context prompting with few examples drive open-ended tasks like captioning and visual Question Answering?

Key findings

• Flamingo achieves new state-of-the-art performance in few-shot learning on a wide range of 16 multimodal tasks.
• On six tasks, Flamingo matches or surpasses fine-tuned SotA using only 32 task-specific examples.
• Model scale and number of shots improve few-shot performance, with larger models better leveraging more shots.
• The architecture with gated cross-attention and the Perceiver Resampler enables conditioning of a frozen LM on interleaved visuals while maintaining training stability.
• Fine-tuning Flamingo on more data sets new SotA on several tasks (VQAv2, VATEX, VizWiz, MSRVTTQA, HatefulMemes).