[论文解读] mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models
该论文提出了 mPLUG-Owl3,一个具备 Hyper Attention blocks 的轻量级多模态大语言模型,能够高效地将视觉和语言统一用于长图像序列,在同等规模的模型中于单图像、多图像和视频基准上实现最先进的结果,并为长视觉上下文引入 Distractor Resistance。
Multi-modal Large Language Models (MLLMs) have demonstrated remarkable capabilities in executing instructions for a variety of single-image tasks. Despite this progress, significant challenges remain in modeling long image sequences. In this work, we introduce the versatile multi-modal large language model, mPLUG-Owl3, which enhances the capability for long image-sequence understanding in scenarios that incorporate retrieved image-text knowledge, interleaved image-text, and lengthy videos. Specifically, we propose novel hyper attention blocks to efficiently integrate vision and language into a common language-guided semantic space, thereby facilitating the processing of extended multi-image scenarios. Extensive experimental results suggest that mPLUG-Owl3 achieves state-of-the-art performance among models with a similar size on single-image, multi-image, and video benchmarks. Moreover, we propose a challenging long visual sequence evaluation named Distractor Resistance to assess the ability of models to maintain focus amidst distractions. Finally, with the proposed architecture, mPLUG-Owl3 demonstrates outstanding performance on ultra-long visual sequence inputs. We hope that mPLUG-Owl3 can contribute to the development of more efficient and powerful multimodal large language models.
研究动机与目标
- 解决多模态 LLMs 在长图像序列理解方面的挑战。
- 开发一种轻量级的跨注意力机制,在扩展时间范围的同时保留视觉细节。
- 实现对交错的视觉-语言输入和超长视觉序列的高效处理。
- 在单图像、multi-image、和视频基准上评估性能,并引入长上下文干扰项测试。
提出的方法
- 引入 Hyper Attention Transformer Block (HATB),在并行中扩展自注意力与跨注意力。
- 使用 Multimodal-Interleaved Rotary Position Embedding (MI-Rope) 来对交错序列中的视觉位置进行编码。
- 采用自适应门控机制,根据文本上下文融合视觉与文本特征。
- 共享一个从语言模型初始化的视觉 KV 投影,以保留视觉信息。
- 分三阶段训练:图文预训练、多图像预训练(包括交错、文本丰富和视频数据),以及使用指令数据的有监督微调。

实验结果
研究问题
- RQ1如何在多模态 LLM 中高效融合长时间交错的图像-文本序列,而不产生过高的内存或延迟?
- RQ2与先前的跨注意力方法相比,Hyper Attention 是否在单图像、跨图像和视频任务中提供稳健的性能?
- RQ3MI-Rope 与自适应门控对多图像和视频理解性能有何影响?
- RQ4拥有 8B 参数的模型能否在多样化的多模态基准上实现最先进的结果,同时保持高效?
主要发现
- mPLUG-Owl3 在覆盖单图像、多图像、和视频任务的 20 项基准中的 14 项上,在同等规模模型中实现了最先进的性能。
- The Hyper Attention design yields strong single-image results and improves multi-image and video performance over concatenation-based and other cross-attention schemes.
- MI-Rope and adaptive gating contribute significantly to multi-image and video understanding, with four integrated HATB layers providing a balance between performance and efficiency.
- On ultra-long visual sequences and distractor-rich contexts, mPLUG-Owl3 demonstrates robust distractor resistance, outperforming several baselines in long-context evaluations.
- The model shows superior inference speed and memory efficiency for its 8B parameter scale compared to larger peers, aided by the lightweight Hyper Attention blocks.

阅读本论文
在 Nubint 阅读器中打开原文,边读边就任意段落提问。
在 Nubint 中阅读本论文本解读由 AI 根据论文原文生成,引用前请核对原文。