[Paper Review] A Survey on Multimodal Large Language Models

Shukang Yin, Chaoyou Fu+5 more|arXiv (Cornell University)|Jun 23, 2023

Topic ModelingComputer Science85 citations

TL;DR

A survey that organizes and summarizes the progress of Multimodal Large Language Models (MLLMs), detailing core techniques like M-IT, M-ICL, M-CoT, and LAVR, and outlining challenges and directions.

ABSTRACT

Recently, Multimodal Large Language Model (MLLM) represented by GPT-4V has been a new rising research hotspot, which uses powerful Large Language Models (LLMs) as a brain to perform multimodal tasks. The surprising emergent capabilities of MLLM, such as writing stories based on images and OCR-free math reasoning, are rare in traditional multimodal methods, suggesting a potential path to artificial general intelligence. To this end, both academia and industry have endeavored to develop MLLMs that can compete with or even better than GPT-4V, pushing the limit of research at a surprising speed. In this paper, we aim to trace and summarize the recent progress of MLLMs. First of all, we present the basic formulation of MLLM and delineate its related concepts, including architecture, training strategy and data, as well as evaluation. Then, we introduce research topics about how MLLMs can be extended to support more granularity, modalities, languages, and scenarios. We continue with multimodal hallucination and extended techniques, including Multimodal ICL (M-ICL), Multimodal CoT (M-CoT), and LLM-Aided Visual Reasoning (LAVR). To conclude the paper, we discuss existing challenges and point out promising research directions. In light of the fact that the era of MLLM has only just begun, we will keep updating this survey and hope it can inspire more research. An associated GitHub link collecting the latest papers is available at https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models.

Motivation & Objective

• Define and formalize Multimodal Large Language Models (MLLMs) and related concepts.
• Provide a comprehensive taxonomy of MLLMs across four main categories: Multimodal Instruction Tuning (M-IT), Multimodal In-Context Learning (M-ICL), Multimodal Chain-of-Thought (M-CoT), and LLM-Aided Visual Reasoning (LAVR).
• Summarize key techniques, data strategies, bridging methods, and evaluation approaches used in MLLMs.
• Highlight challenges and propose promising research directions for the field.

Proposed method

• Present a formal formulation of MLLMs and their instruction/interaction paradigms.
• Categorize existing work into four genres (M-IT, M-ICL, M-CoT, LAVR) and discuss their architectural and data requirements.
• Describe data collection methods (benchmark adaptation, self-instruction, hybrid composition) and modality bridging (learnable interfaces vs. expert models).
• Explain alignment pre-training and multimodal data construction for M-IT, including instruction templates and evaluation methodologies.
• Summarize learning paradigms (finetuning, few-shot, zero-shot) and generation patterns (infilling vs. predicting) in M-CoT and LAVR, and discuss evaluation frameworks (closed-set vs. open-set).

Figure 1 : Comparisons of three typical learning paradigms. The image is from [ 16 ] .

Experimental results

Research questions

• RQ1 What are the core paradigms enabling multimodal reasoning in LLM-based systems?
• RQ2 How do data construction and modality bridging shape the performance of MLLMs across M-IT, M-ICL, M-CoT, and LAVR?
• RQ3 What evaluation strategies are appropriate for multimodal instruction tuning and visual reasoning systems?
• RQ4 What are the main challenges and potential directions for advancing MLLMs toward more general capabilities?

Key findings

• MLLMs leverage four main techniques: Multimodal Instruction Tuning (M-IT), Multimodal In-Context Learning (M-ICL), Multimodal Chain-of-Thought (M-CoT), and LLM-Aided Visual Reasoning (LAVR).
• Data construction for M-IT includes benchmark adaptation, self-instruction, and hybrid composition to create multimodal instruction data.
• Modality bridging is typically achieved via learnable interfaces or expert models to translate visual content into text for LLMs.
• Evaluation distinguishes closed-set and open-set tasks, with additional benchmarks and human/AI scoring approaches for open-ended multimodal tasks.
• The survey highlights multiple future directions and ongoing challenges in MLLMs, including scalability, alignment, robustness, and multimodal reasoning capabilities.