[Paper Review] Language Is Not All You Need: Aligning Perception with Language Models

Shaohan Huang, Dong Li+16 more|arXiv (Cornell University)|Feb 27, 2023

Multimodal Machine Learning ApplicationsComputer Science164 citations

TL;DR

Kosmos-1 is a multimodal large language model trained from scratch on web-scale text, image, and interleaved multimodal data to perform zero-shot and few-shot reasoning across language, perception, and vision without fine-tuning.

ABSTRACT

A big convergence of language, multimodal perception, action, and world modeling is a key step toward artificial general intelligence. In this work, we introduce Kosmos-1, a Multimodal Large Language Model (MLLM) that can perceive general modalities, learn in context (i.e., few-shot), and follow instructions (i.e., zero-shot). Specifically, we train Kosmos-1 from scratch on web-scale multimodal corpora, including arbitrarily interleaved text and images, image-caption pairs, and text data. We evaluate various settings, including zero-shot, few-shot, and multimodal chain-of-thought prompting, on a wide range of tasks without any gradient updates or finetuning. Experimental results show that Kosmos-1 achieves impressive performance on (i) language understanding, generation, and even OCR-free NLP (directly fed with document images), (ii) perception-language tasks, including multimodal dialogue, image captioning, visual question answering, and (iii) vision tasks, such as image recognition with descriptions (specifying classification via text instructions). We also show that MLLMs can benefit from cross-modal transfer, i.e., transfer knowledge from language to multimodal, and from multimodal to language. In addition, we introduce a dataset of Raven IQ test, which diagnoses the nonverbal reasoning capability of MLLMs.

Motivation & Objective

• Motivate the need for models that align multimodal perception with language models to pursue artificial general intelligence.
• Develop a model that can perceive general modalities, follow instructions, and learn in-context without fine-tuning.
• Show that language models can serve as a universal task interface across modalities.
• Demonstrate cross-modal transfer benefits between language-only and multimodal capabilities.
• Provide benchmarks and a new Raven IQ-style nonverbal reasoning dataset for MLLMs.

Proposed method

• Train Kosmos-1 from scratch on web-scale multimodal corpora including interleaved text-and-image data, image-caption pairs, and text-only data.
• Use a Transformer-based causal language model as the core interface with embedded multimodal inputs.
• Adopt Magneto backbones and xPos relative position encoding to improve stability and long-context modeling.
• Pretrain with next-token prediction on mixed modalities, keeping discrete token losses for training.
• Conduct language-only instruction tuning to improve instruction-following and transfer to multimodal tasks.

Experimental results

Research questions

• RQ1 Can a Multimodal Large Language Model (MLLM) align perception with a language model to perform both language and vision tasks without fine-tuning?
• RQ2 To what extent can cross-modal transfer improve language and perception-language tasks, and vice versa?
• RQ3 How well does an MLLM handle nonverbal reasoning, OCR-free tasks, and perception-based reasoning compared with text-only models?

Key findings

• Kosmos-1 demonstrates zero-shot and few-shot capabilities across language, perception-language, and vision tasks without gradient updates.
• The model benefits from cross-modal transfer, with language and multimodal capabilities supporting each other.
• A Raven IQ-style nonverbal reasoning benchmark shows Kosmos-1 can perform zero-shot nonverbal reasoning, indicating abstract pattern recognition in vision-text contexts.
• OCR-free tasks such as rendered text and web-page understanding are feasible with Kosmos-1, without external tools.
• Multimodal chain-of-thought prompting improves performance on perception-language tasks by generating intermediate rationale before final answers.