[論文レビュー] Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models

Chi Chen, Ruoyu Qin+5 more|arXiv (Cornell University)|Aug 25, 2023

Multimodal Machine Learning Applications被引用数 10

ひとこと要約

PVIT は MLLMs に地域レベルのビジョンエンコーダを追加し、地域中心の指示データ生成方式を導入することで、より細かな画像理解と卓越したマルチモーダル推論を実現します。

ABSTRACT

Recently, Multimodal Large Language Models (MLLMs) that enable Large Language Models (LLMs) to interpret images through visual instruction tuning have achieved significant success. However, existing visual instruction tuning methods only utilize image-language instruction data to align the language and image modalities, lacking a more fine-grained cross-modal alignment. In this paper, we propose Position-enhanced Visual Instruction Tuning (PVIT), which extends the functionality of MLLMs by integrating an additional region-level vision encoder. This integration promotes a more detailed comprehension of images for the MLLM. In addition, to efficiently achieve a fine-grained alignment between the vision modules and the LLM, we design multiple data generation strategies to construct an image-region-language instruction dataset. Finally, we present both quantitative experiments and qualitative analysis that demonstrate the superiority of the proposed model. Code and data will be released at https://github.com/PVIT-official/PVIT.

研究の動機と目的

• Motivation to achieve fine-grained cross-modal alignment beyond image-level supervision in MLLMs.
• Extend MLLMs with region-level understanding via an additional region encoder.
• Develop data generation strategies to create region-level image-region-language instructions.
• Evaluate PVIT on object recognition and multimodal reasoning, plus human-centric FineEval assessment.

提案手法

• Integrate a region-level vision encoder (RegionCLIP-based) with an LLM, using image regions as input alongside image and text.
• Use a two-stage training: stage 1 align region features to LLM embeddings via a linear projection, stage 2 fine-tune end-to-end for region-based instructions.
• Construct region-level instruction data via (a) dataset conversion of GQA/VCR, (b) task-specific data generation with ChatGPT, (c) general data generation with rich descriptions and grounding annotations.
• Train using frozen image/region encoders and trainable LLM plus projection layer; follow with further fine-tuning on region-level instructions.
• Leverage region-level supervision to improve object-region comprehension and spatial reasoning.

実験結果

リサーチクエスチョン

• RQ1 Can region-level vision encoders be effectively integrated into MLLMs without disrupting existing capabilities?
• RQ2 Does region-level instruction data improve fine-grained spatial understanding and region-based question answering?
• RQ3 What data generation strategies yield diverse and high-quality region-level instructions for training PVIT?
• RQ4 How does PVIT perform on recognition and multimodal reasoning tasks compared to image-level only baselines?

主な発見

Method	COCO	GQA
LLaVA [16]	40.04	46.82
Shikra [3]	53.91	54.81
GPT4RoI [34]	64.01	52.64
PVIT (Ours)	64.53	55.77

• PVIT achieves the best GQA accuracy among compared models in multimodal reasoning (55.77 on GQA).
• PVIT outperforms baselines LLaVA, Shikra, and GPT4RoI on multimodal reasoning (GQA) and achieves competitive/objective recognition on COCO.
• Human evaluation (FineEval) shows PVIT consistently ranks higher than baselines in fine-grained spatial instruction following, with a minor exception in object counting vs. Shikra.
• Ablation indicates region representations (_region features_) substantially improve performance over textual region coordinates.
• Two-stage training effectively aligns region features to the LLM without overhauling pre-trained encoders.