[Paper Review] NExT-GPT: Any-to-Any Multimodal LLM

Shengqiong Wu, Fei Hao+3 more|arXiv (Cornell University)|Sep 11, 2023

Topic ModelingComputer Science94 citations

TL;DR

NExT-GPT is an end-to-end any-to-any multimodal LLM that can accept and generate content across text, images, videos, and audio by linking an LLM with multimodal encoders/diffusion decoders and lightweight projections, plus modality-switching instruction tuning (MosIT).

ABSTRACT

While recently Multimodal Large Language Models (MM-LLMs) have made exciting strides, they mostly fall prey to the limitation of only input-side multimodal understanding, without the ability to produce content in multiple modalities. As we humans always perceive the world and communicate with people through various modalities, developing any-to-any MM-LLMs capable of accepting and delivering content in any modality becomes essential to human-level AI. To fill the gap, we present an end-to-end general-purpose any-to-any MM-LLM system, NExT-GPT. We connect an LLM with multimodal adaptors and different diffusion decoders, enabling NExT-GPT to perceive inputs and generate outputs in arbitrary combinations of text, images, videos, and audio. By leveraging the existing well-trained highly-performing encoders and decoders, NExT-GPT is tuned with only a small amount of parameter (1%) of certain projection layers, which not only benefits low-cost training and also facilitates convenient expansion to more potential modalities. Moreover, we introduce a modality-switching instruction tuning (MosIT) and manually curate a high-quality dataset for MosIT, based on which NExT-GPT is empowered with complex cross-modal semantic understanding and content generation. Overall, our research showcases the promising possibility of building an AI agent capable of modeling universal modalities, paving the way for more human-like AI research in the community. Project page: https://next-gpt.github.io/

Motivation & Objective

• Address the gap of MM-LLMs that only understand inputs but cannot output across multiple modalities.
• Develop an end-to-end any-to-any MM-LLM that handles text, images, videos, and audio in both input and output.
• Leverage off-the-shelf encoders/decoders to minimize training cost and enable easy modality expansion.
• Introduce modality-switching instruction tuning (MosIT) with a high-quality dataset to enhance cross-modal reasoning and generation.

Proposed method

• Three-tier architecture: multimodal encoding with off-the-shelf encoders and a projection layer to language space; LLM-based understanding and reasoning; multimodal decoding via diffusion decoders conditioned on modality signals.
• Use ImageBind-based or other encoders to map inputs into language-like representations; keep encoders/decoders frozen and only train input/output projection layers (approximately 1% of parameters).
• LLM (Vicuna 7B) outputs textual tokens and modality signal tokens that instruct decoders whether and what to generate in each modality.
• Modal signals are defined as specific tokens (e.g., <IMG_i>, <AUD_i>, <VID_i>) that route representations to corresponding diffusion decoders for content generation.
• Lightweight alignment: encoding-side LLM-centric multimodal alignment trained with caption-like objectives; decoding-side instruction-following alignment aligns diffusion condition encoders with LLM outputs.
• MosIT data: a 5K high-quality multimodal instruction-tuning dataset crafted with templates and GPT-4 to cover complex cross-modal instructions and multi-turn dialogues.

Experimental results

Research questions

• RQ1 Can an LLM-centric end-to-end system understand and generate content across any combination of text, images, video, and audio?
• RQ2 What training strategy enables efficient cross-modal alignment with minimal parameter updates?
• RQ3 Does modality-switching instruction tuning improve cross-modal reasoning and generation quality across diverse modality conversions?

Key findings

• NExT-GPT achieves competitive or superior generation quality across several text-to-X and X-to-text tasks compared with baselines (e.g., text-to-image: NExT-GPT 11.28 FID on COCO-caption vs CogVideo 27.10; 11.26 for CoDi).
• Text-to-audio: NExT-GPT FD 23.58 and IS 8.35 on AudioCaps, comparing favorably with several baselines.
• Text-to-video: NExT-GPT FD 13.04 and CLIPSIM 0.3085 on MSR-VTT, showing strong performance among diffusion-based systems.
• Image-to-text (captioning) on COCO-caption: NExT-GPT B@4 44.3 and CIDEr 156.7, exceeding several baselines.
• Audio-to-text: NExT-GPT B@4 58.4 and METEOR 38.5 on AudioCaps, outperforming many alternatives.
• Video-to-text on MSR-VTT: NExT-GPT CIDEr 0.802, indicating strong video captioning capability.