[Paper Review] To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuning

Junke Wang, Lingchen Meng+4 more|arXiv (Cornell University)|Nov 13, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes LVIS-Instruct4V, a 220K-sample visual instruction dataset generated by prompting GPT-4V with images and bounding box annotations from the LVIS dataset. By leveraging GPT-4V’s direct visual understanding, the method produces fine-grained, context-aware, and visually aligned instruction-answer pairs, significantly improving LLaVA-1.5’s performance across benchmarks, including SOTA results on MME (+43.6) and MM-Vet (+5.6%).

ABSTRACT

Existing visual instruction tuning methods typically prompt large language models with textual descriptions to generate instruction-following data. Despite the promising performance achieved, these descriptions are derived from image annotations, which are oftentimes coarse-grained. Furthermore, the instructions might even contradict the visual content without observing the entire visual context. To address this challenge, we introduce a fine-grained visual instruction dataset, LVIS-Instruct4V, which contains 220K visually aligned and context-aware instructions produced by prompting the powerful GPT-4V with images from LVIS. Through experimental validation and case studies, we demonstrate that high-quality visual instructional data could improve the performance of LLaVA-1.5, a state-of-the-art large multimodal model, across a wide spectrum of benchmarks by clear margins. Notably, by simply replacing the LLaVA-Instruct with our LVIS-Instruct4V, we achieve better results than LLaVA on most challenging LMM benchmarks, e.g., LLaVA$^w$ (76.7 vs. 70.7) and MM-Vet (40.2 vs. 35.4). We release our data and model at https://github.com/X2FD/LVIS-INSTRUCT4V.

Motivation & Objective

• To address the limitations of existing visual instruction tuning methods that rely on coarse textual descriptions from image annotations.
• To improve the accuracy and visual grounding of instruction-following data by enabling large multimodal models to directly observe images during data generation.
• To develop a high-quality, fine-grained visual instruction dataset that enhances the performance of multimodal language models on complex benchmarks.
• To demonstrate that GPT-4V can generate superior instruction data compared to text-only prompting, leading to measurable gains in downstream multimodal reasoning.
• To validate that replacing standard instruction data with LVIS-Instruct4V leads to state-of-the-art performance on challenging LMM benchmarks.

Proposed method

• Utilize GPT-4V, a vision-language model with direct visual input, to generate instruction-answer pairs based on raw images and their bounding box annotations from the LVIS dataset.
• Design a two-stage prompting pipeline: first, generate contextually aware visual dialogues; second, produce high-quality image descriptions grounded in visual details.
• Curate 220,000 visually aligned and fine-grained instruction-answer pairs by prompting GPT-4V with both images and their localized annotations.
• Fine-tune LLaVA-1.5 using the LVIS-Instruct4V dataset as the sole instruction tuning data, maintaining the same model architecture and training procedure.
• Enhance performance further by mixing LVIS-Instruct4V with LLaVA-Instruct during training, improving generalization and robustness.
• Conduct ablation studies to evaluate the impact of fine-tuning the language model, training schedule length, and data mixing strategies.

Experimental results

Research questions

• RQ1 Can GPT-4V generate more accurate and context-aware visual instruction data when given direct access to images compared to text-only prompting?
• RQ2 To what extent does using GPT-4V-generated instruction data improve the performance of multimodal models on fine-grained visual reasoning benchmarks?
• RQ3 How does LVIS-Instruct4V compare to existing instruction tuning datasets like LLaVA-Instruct in terms of visual grounding and model generalization?
• RQ4 What is the impact of data mixing (LVIS-Instruct4V + LLaVA-Instruct) on model performance across diverse multimodal benchmarks?
• RQ5 Does the quality of instruction data correlate with improvements in challenging benchmarks such as MME and MM-Vet?

Key findings

• LLaVA-1.5 fine-tuned on LVIS-Instruct4V alone achieves 76.7 on LLaVA w and 40.2 on MM-Vet, outperforming the original LLaVA model (70.7 and 35.4).
• The 13B version of the model trained on LVIS-Instruct4V achieves 80.7 on VQAv2, surpassing LLaVA-1.5’s 80.0 score.
• Mixing LVIS-Instruct4V with LLaVA-Instruct yields state-of-the-art results on 9 out of 11 benchmarks, including 1574.9 on MME (a 43.6-point gain over LLaVA-1.5).
• The model trained on LVIS-Instruct4V demonstrates superior scene text recognition, correctly identifying two texts in an image where LLaVA-1.5 missed one.
• The model correctly identifies object relationships (e.g., bowl positions) in fine-grained visual comparisons, while LLaVA-1.5 makes incorrect assignments.
• Ablation studies confirm that even with a frozen language model, LVIS-Instruct4V improves performance on MME (1544.3 vs. 1531.3), and fine-tuning the model further boosts it to 1581.3.