[논문 리뷰] A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks

Jiaqi Wang, Hanqi Jiang외 22명|arXiv (Cornell University)|2024. 08. 02.

Topic Modeling인용 수 16

한 줄 요약

A systematic survey of Multimodal Large Language Models (MLLMs) detailing architectures, tasks, performance, challenges, and future directions.

ABSTRACT

In an era defined by the explosive growth of data and rapid technological advancements, Multimodal Large Language Models (MLLMs) stand at the forefront of artificial intelligence (AI) systems. Designed to seamlessly integrate diverse data types-including text, images, videos, audio, and physiological sequences-MLLMs address the complexities of real-world applications far beyond the capabilities of single-modality systems. In this paper, we systematically sort out the applications of MLLM in multimodal tasks such as natural language, vision, and audio. We also provide a comparative analysis of the focus of different MLLMs in the tasks, and provide insights into the shortcomings of current MLLMs, and suggest potential directions for future research. Through these discussions, this paper hopes to provide valuable insights for the further development and application of MLLM.

연구 동기 및 목표

• Assess the scope and impact of MLLMs across text, image, video, and audio modalities.
• Summarize core architectures and components used in MLLMs, including encoders, fusion mechanisms, and decoders.
• Evaluate how MLLMs perform on image, video, and audio tasks and identify their strengths and limitations.
• Identify current challenges and outline promising directions for future research and applications.

제안 방법

• Describe the three main MLLM components: multimodal input encoder, feature fusion mechanism, and multimodal output decoder.
• Explain fusion strategies (early, intermediate, late, joint) and how they integrate modalities with pre-trained LLMs.
• Present representative models (e.g., MiniGPT-4, InstructBLIP) and their architecture, datasets, and training regimes.
• Discuss multimodal feature projection and how image, text, and audio features are mapped into a shared space for LLM processing.
• Review two-stage training paradigms and instruction tuning used to align vision-language capabilities with LLMs.
• Classify tasks into image understanding and generation, and summarize task-specific advancements.

실험 결과

연구 질문

• RQ1 What are the core architectural components enabling multimodal integration in LLMs?
• RQ2 How do fusion strategies influence performance across vision and audio tasks in MLLMs?
• RQ3 What are the current strengths and limitations of MLLMs in image understanding and generation?
• RQ4 What datasets, training regimes, and model alignments drive effectiveness and reliability of MLLMs?
• RQ5 What future directions and challenges are highlighted for advancing MLLMs?

주요 결과

• MLLMs demonstrate strong capabilities in integrating text with visual and auditory data to enhance understanding and generation.
• Image understanding and image generation are central tasks where MLLMs show notable progress through multimodal fusion and instruction-following abilities.
• Fusion strategies (early, intermediate, late, joint) play crucial roles in how effectively modalities are combined for downstream tasks.
• Representative models like MiniGPT-4 and InstructBLIP illustrate practical architectures and training paradigms for aligning vision-language capabilities with LLMs.
• Current shortcomings include coherence of language outputs, data diversity, and the need for scalable, efficient training and evaluation frameworks.
• Future directions emphasize improved multimodal alignment, better benchmarks, and more robust, adaptable cross-modal reasoning.