[Paper Review] Kosmos-2: Grounding Multimodal Large Language Models to the World

Zhiliang Peng, Wenhui Wang+5 more|arXiv (Cornell University)|Jun 26, 2023

Multimodal Machine Learning Applications130 citations

TL;DR

Kosmos-2 is a grounded multimodal LLM that links text spans to image regions via location tokens, built with the GrIT dataset, and evaluated on grounding, referring, perception-language, and language tasks.

ABSTRACT

We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enabling new capabilities of perceiving object descriptions (e.g., bounding boxes) and grounding text to the visual world. Specifically, we represent refer expressions as links in Markdown, i.e., ``[text span](bounding boxes)'', where object descriptions are sequences of location tokens. Together with multimodal corpora, we construct large-scale data of grounded image-text pairs (called GrIT) to train the model. In addition to the existing capabilities of MLLMs (e.g., perceiving general modalities, following instructions, and performing in-context learning), Kosmos-2 integrates the grounding capability into downstream applications. We evaluate Kosmos-2 on a wide range of tasks, including (i) multimodal grounding, such as referring expression comprehension, and phrase grounding, (ii) multimodal referring, such as referring expression generation, (iii) perception-language tasks, and (iv) language understanding and generation. This work lays out the foundation for the development of Embodiment AI and sheds light on the big convergence of language, multimodal perception, action, and world modeling, which is a key step toward artificial general intelligence. Code and pretrained models are available at https://aka.ms/kosmos-2.

Motivation & Objective

• Motivate grounding as a foundational capability for multimodal LLMs to reduce ambiguity and enable visual answers.
• Introduce GrIT, a large-scale grounded image-text dataset linking phrases to bounding boxes.
• Extend Kosmos-1 with grounding and referring to enable region-based input/output and visual grounding of text.
• Demonstrate Kosmos-2’s effectiveness on grounding, referring, perception-language, and language tasks.

Proposed method

• Represent refer expressions as hyperlinks in Markdown linking text spans to sequences of location tokens.
• Construct GrIT by extracting noun chunks and referring expressions from captions and grounding them to image regions using detectors (e.g., GLIP) and spaCy-based parsing.
• Discretize bounding box coordinates into location tokens and append them to the corresponding text spans in a unified <grounding> format.
• Train Kosmos-2 as a Transformer-based causal language model initialized from Kosmos-1, with grounding-specific data and a next-token objective.
• In instruction tuning, combine vision-language and language-only instruction data with grounded data from GrIT to align model outputs with grounded prompts.

Experimental results

Research questions

• RQ1 Can Kosmos-2 ground text to specific image regions and output bounding boxes as part of its responses?
• RQ2 How does grounding affect performance on phrase grounding and referring expression tasks compared to prior MLLMs?
• RQ3 Can Kosmos-2 perform grounding-enabled downstream tasks such as grounded image captioning and grounded VQA?
• RQ4 How do grounding and referring capabilities interact with standard perception-language and language tasks?

Key findings

• Kosmos-2 achieves strong grounding performance, including phrase grounding and referring expression comprehension, and surpasses several baselines on grounding benchmarks.
• It demonstrates effective referring expression generation, with zero-shot and few-shot improvements on RefCOCOg.
• Grounded inputs enable new downstream tasks like grounded image captioning and grounded visual question answering without sacrificing core language and vision-language abilities.
• On perception-language tasks (image captioning and VQA), Kosmos-2 remains competitive with Kosmos-1, showing no large degradation from grounding integration.
• On standard language tasks, Kosmos-2 shows comparable performance to Kosmos-1 with some metrics improving (e.g., BoolQ, COPA) and others remaining stable.