[Paper Review] mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality

Qinghao Ye, Haiyang Xu+16 more|arXiv (Cornell University)|Apr 27, 2023

Multimodal Machine Learning ApplicationsComputer Science167 citations

TL;DR

mPLUG-Owl introduces a modular training paradigm that uses a frozen language model plus a trainable visual knowledge module and a visual abstractor, enabling multimodal understanding and multi-turn dialogue, evaluated on OwlEval.

ABSTRACT

Large language models (LLMs) have demonstrated impressive zero-shot abilities on a variety of open-ended tasks, while recent research has also explored the use of LLMs for multi-modal generation. In this study, we introduce mPLUG-Owl, a novel training paradigm that equips LLMs with multi-modal abilities through modularized learning of foundation LLM, a visual knowledge module, and a visual abstractor module. This approach can support multiple modalities and facilitate diverse unimodal and multimodal abilities through modality collaboration. The training paradigm of mPLUG-Owl involves a two-stage method for aligning image and text, which learns visual knowledge with the assistance of LLM while maintaining and even improving the generation abilities of LLM. In the first stage, the visual knowledge module and abstractor module are trained with a frozen LLM module to align the image and text. In the second stage, language-only and multi-modal supervised datasets are used to jointly fine-tune a low-rank adaption (LoRA) module on LLM and the abstractor module by freezing the visual knowledge module. We carefully build a visually-related instruction evaluation set OwlEval. Experimental results show that our model outperforms existing multi-modal models, demonstrating mPLUG-Owl's impressive instruction and visual understanding ability, multi-turn conversation ability, and knowledge reasoning ability. Besides, we observe some unexpected and exciting abilities such as multi-image correlation and scene text understanding, which makes it possible to leverage it for harder real scenarios, such as vision-only document comprehension. Our code, pre-trained model, instruction-tuned models, and evaluation set are available at https://github.com/X-PLUG/mPLUG-Owl. The online demo is available at https://www.modelscope.cn/studios/damo/mPLUG-Owl.

Motivation & Objective

• Motivate enabling multimodal abilities in LLMs without fully retraining large language models.
• Propose a modular architecture combining a vision foundation model, a visual knowledge module, and a visual abstractor.
• Develop a two-stage training paradigm to align images with text while preserving LLM generation capability.
• Demonstrate improved unimodal and multimodal instruction understanding and multi-turn dialogue through joint instruction tuning.

Proposed method

• Use a vision foundation model f_V to extract visual features.
• Introduce a visual abstractor f_K to summarize visual features into learnable tokens.
• Freeze the language foundation model f_L while training the visual components to align image-text representations.
• Stage 1: train visual knowledge and abstractor with frozen LLM using image-caption pairs.
• Stage 2: joint instruction tuning with language-only and multi-modal data by freezing f_V and training a LoRA on f_L and f_K.

Figure 1 : Comparison between different training paradigms. All of these methods are trained in a two-stage fashion. Stage 1 stands for pre-training and Stage 2 represents instruction tuning.

Experimental results

Research questions

• RQ1 Can a modular vision-language setup align visual knowledge with a frozen LLM to enable multimodal understanding?
• RQ2 Does two-stage training with multimodal and text-only data improve unimodal and multimodal instruction following compared to baselines?
• RQ3 What emergent abilities arise from modular multimodal training (e.g., multi-image correlation, scene text understanding, multilingual dialogue)?

Key findings

• mPLUG-Owl outperforms baselines like MiniGPT-4 and LLaVA on OwlEval in instruction understanding and visual tasks.
• The two-stage training scheme with multimodal pretraining and joint instruction tuning yields the best performance.
• Joint multimodal and text-only instruction data enhance both knowledge transfer and reasoning abilities.
• Ablations show multimodal data during instruction tuning improves visual knowledge alignment and text-only task performance.
• Qualitative analyses reveal emergent abilities such as multi-image correlation and multilingual conversation.

Figure 2 : Our training paradigm and model overview.