[Paper Review] LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models

Feng Li, Renrui Zhang+6 more|arXiv (Cornell University)|Jul 10, 2024

Simulation and Modeling Applications21 citations

TL;DR

LLaVA-NeXT-Interleave unifies multi-image, video, 3D, and single-image tasks into an interleaved data format, training with M4-Instruct and evaluating on the new LLaVA-Interleave Bench to achieve state-of-the-art results across M4 scenarios while preserving single-image performance.

ABSTRACT

Visual instruction tuning has made considerable strides in enhancing the capabilities of Large Multimodal Models (LMMs). However, existing open LMMs largely focus on single-image tasks, their applications to multi-image scenarios remains less explored. Additionally, prior LMM research separately tackles different scenarios, leaving it impossible to generalize cross scenarios with new emerging capabilities. To this end, we introduce LLaVA-NeXT-Interleave, which simultaneously tackles Multi-image, Multi-frame (video), Multi-view (3D), and Multi-patch (single-image) scenarios in LMMs. To enable these capabilities, we regard the interleaved data format as a general template and compile the M4-Instruct dataset with 1,177.6k samples, spanning 4 primary domains with 14 tasks and 41 datasets. We also curate the LLaVA-Interleave Bench to comprehensively evaluate the multi-image performance of LMMs. Through extensive experiments, LLaVA-NeXT-Interleave achieves leading results in multi-image, video, and 3D benchmarks, while maintaining the performance of single-image tasks. Besides, our model also exhibits several emerging capabilities, e.g., transferring tasks across different settings and modalities. Code is available at https://github.com/LLaVA-VL/LLaVA-NeXT

Motivation & Objective

• Motivate and enable a single LMM to handle multi-image, video, 3D, and single-image tasks (M4).
• Propose an interleaved image-text data template to unify diverse tasks under one framework.
• Create and curate the M4-Instruct dataset and LLaVA-Interleave Bench for training and evaluation across domains.

Proposed method

• Adopt the LLaVA-NeXT-Image architecture with vision encoder, intermediate projector, and LLM core.
• Introduce three training techniques: (1) continue from a strong single-image model, (2) mixed interleaved data formats (in-front vs interleaved), (3) joint training on four data scenarios (multi-image, multi-frame, multi-view, multi-patch).
• Construct M4-Instruct with 1,177.6K samples spanning 14 tasks and 41 datasets across M4 domains; annotate new tasks via GPT-4V.
• Develop LLaVA-Interleave Bench with 13 tasks and 17K instances, split into in-domain and out-domain evaluations.
• Evaluate across multi-image, video, and 3D benchmarks, plus maintain single-image performance.

Experimental results

Research questions

• RQ1 Can a single LMM trained on interleaved multi-image data perform competitively on multi-image tasks and generalize to video and 3D scenarios?
• RQ2 Does interleaved data format enable cross-task transfer and emerging capabilities across modalities?
• RQ3 How does initializing from a strong single-image checkpoint impact multi-image fine-tuning performance?
• RQ4 What is the impact of input token placement (in-front vs interleaved) and mixed-format training on robustness and task performance?

Key findings

• LLaVA-NeXT-Interleave achieves state-of-the-art results on multi-image benchmarks while preserving single-image performance across sizes (0.5B, 7B, 14B).
• The interleaved data template and joint M4-Instruct training enable cross-task transfer like single-image-to-multi-image reasoning and image-to-video task transfer.
• Adding video and multi-image data via mixed formats improves overall metrics and robustness across tasks.
• The model demonstrates emerging capabilities such as transferring tasks across settings and modalities (e.g., from spotting differences to videos, or generating Twitter posts from video).
• LLaVA-Interleave Bench provides evaluation coverage with in-domain and out-domain tasks, highlighting generalization to unseen multi-image scenarios.