[Paper Review] Emu3: Next-Token Prediction is All You Need

Xinlong Wang, Xiaosong Zhang+23 more|arXiv (Cornell University)|Sep 27, 2024

Medical Imaging and Pathology Studies4 citations

TL;DR

Emu3 introduces a unified multimodal foundation model trained exclusively via next-token prediction, tokenizing text, images, and video into a shared discrete space. It achieves state-of-the-art performance across generation and perception tasks—surpassing models like SDXL and LLaVA-1.6—without relying on diffusion or compositional architectures, demonstrating that next-token prediction alone can enable general multimodal intelligence.

ABSTRACT

While next-token prediction is considered a promising path towards artificial general intelligence, it has struggled to excel in multimodal tasks, which are still dominated by diffusion models (e.g., Stable Diffusion) and compositional approaches (e.g., CLIP combined with LLMs). In this paper, we introduce Emu3, a new suite of state-of-the-art multimodal models trained solely with next-token prediction. By tokenizing images, text, and videos into a discrete space, we train a single transformer from scratch on a mixture of multimodal sequences. Emu3 outperforms several well-established task-specific models in both generation and perception tasks, surpassing flagship models such as SDXL and LLaVA-1.6, while eliminating the need for diffusion or compositional architectures. Emu3 is also capable of generating high-fidelity video via predicting the next token in a video sequence. We simplify complex multimodal model designs by converging on a singular focus: tokens, unlocking great potential for scaling both during training and inference. Our results demonstrate that next-token prediction is a promising path towards building general multimodal intelligence beyond language. We open-source key techniques and models to support further research in this direction.

Motivation & Objective

• To investigate whether next-token prediction alone can achieve state-of-the-art performance in multimodal tasks, challenging the dominance of diffusion and compositional models.
• To eliminate architectural complexity by unifying vision, language, and video under a single next-token prediction objective.
• To demonstrate that a single transformer trained from scratch on multimodal sequences can match or exceed specialized models in both generation and perception.
• To enable scalable training and inference by focusing solely on token-level prediction, simplifying model design.

Proposed method

• Images, text, and videos are tokenized into a shared discrete space using a learnable tokenizer, enabling joint modeling.
• A single transformer decoder is trained from scratch on a mixture of multimodal sequences, with next-token prediction as the sole objective.
• The model processes sequences of tokens from different modalities in a unified autoregressive manner, predicting the next token in the sequence.
• Video generation is achieved by autoregressively predicting tokens corresponding to video frames in sequence, enabling high-fidelity video synthesis.
• The training data comprises diverse multimodal sequences, including text-image, text-video, and image-video pairs, enabling zero-shot generalization.
• The architecture avoids diffusion, contrastive pretraining, or modular fusion, relying solely on next-token prediction for all capabilities.

Experimental results

Research questions

• RQ1 Can next-token prediction alone achieve state-of-the-art performance in multimodal generation and perception tasks?
• RQ2 Does a unified autoregressive transformer trained exclusively on next-token prediction outperform specialized models like SDXL and LLaVA-1.6?
• RQ3 Can a single model trained on a mixture of text, image, and video sequences generate high-fidelity video through autoregressive token prediction?
• RQ4 To what extent can architectural complexity be reduced by eliminating diffusion and compositional components in multimodal models?
• RQ5 Is next-token prediction sufficient to enable general multimodal intelligence beyond language?

Key findings

• Emu3 outperforms SDXL in text-to-image generation across multiple benchmarks, achieving higher FID and CLIP scores.
• Emu3 surpasses LLaVA-1.6 in visual question answering and image captioning, demonstrating stronger perception capabilities.
• The model generates high-fidelity video by autoregressively predicting tokens in a video sequence, achieving competitive video generation quality.
• Emu3 achieves state-of-the-art results in both generation and perception tasks without using diffusion or contrastive pretraining.
• The model demonstrates strong zero-shot generalization across diverse multimodal tasks, indicating robustness of the next-token prediction objective.
• The elimination of complex components like diffusion heads or modality-specific encoders simplifies training and inference while maintaining or improving performance.