[Paper Review] ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models

Guiming Hardy Chen, Shunian Chen+8 more|arXiv (Cornell University)|Feb 18, 2024

Natural Language Processing Techniques4 citations

TL;DR

This paper introduces ALLaVA, a 3B-parameter vision-language model that achieves competitive performance on 12 benchmarks by training on high-quality, GPT-4V-synthesized data. By leveraging synthetic captions, complex instructions, and detailed answers from GPT-4V, ALLaVA narrows the performance gap between lite and full-sized LVLMs, outperforming larger models on key tasks despite its small size.

ABSTRACT

Large vision-language models (LVLMs) have shown premise in a broad range of vision-language tasks with their strong reasoning and generalization capabilities. However, they require considerable computational resources for training and deployment. This study aims to bridge the performance gap between traditional-scale LVLMs and resource-friendly lite versions by adopting high-quality training data. To this end, we propose a comprehensive pipeline for generating a synthetic dataset. The key idea is to leverage strong proprietary models to generate (i) fine-grained image annotations for vision-language alignment and (ii) complex reasoning visual question-answering pairs for visual instruction fine-tuning, yielding 1.3M samples in total. We train a series of lite VLMs on the synthetic dataset and experimental results demonstrate the effectiveness of the proposed scheme, where they achieve competitive performance on 17 benchmarks among 4B LVLMs, and even perform on par with 7B/13B-scale models on various benchmarks. This work highlights the feasibility of adopting high-quality data in crafting more efficient LVLMs. We name our dataset extit{ALLaVA}, and open-source it to research community for developing better resource-efficient LVLMs for wider usage.

Motivation & Objective

• To close the performance gap between lite and full-scale vision-language models (LVLMs) through high-quality training data.
• To develop a scalable, data-centric approach for generating high-fidelity multimodal training data without relying on human-annotated datasets.
• To evaluate whether synthetic data from GPT-4V can effectively train a lightweight LVLM that matches or exceeds the performance of larger models.
• To open-source a large-scale, high-quality synthetic dataset for LVLM training, including 1.4M samples with detailed captions and instructions.

Proposed method

• Leveraged GPT-4V to generate fine-grained image captions, complex reasoning instructions, and detailed answers for 1.4M images.
• Constructed two distinct synthetic datasets: ALLaVA-Caption-4V for visual-textual alignment and ALLaVA-Instruct-4V for visual instruction tuning.
• Trained a 3B-parameter language model (Phi2) with a vision encoder using a two-stage pipeline: pretraining on alignment data, followed by fine-tuning on instruction data.
• Used a data-centric framework to prioritize data quality over scale, minimizing noisy signals from low-quality captions.
• Evaluated model performance across 12 benchmarks, including MME, GQA, and MM-Vet, comparing against SOTA models like LLaVA-v1.6-34B and GPT-4V.
• Conducted ablation studies to isolate the impact of each synthetic dataset on model performance at different training stages.

Figure 1: Pipeline for scaling up high-quality data. Prompts in the figure are shown for demonstration purpose. See the detailed prompt in Appendix A.1 and A.2 .

Experimental results

Research questions

• RQ1 Can high-quality synthetic data from GPT-4V close the performance gap between lite and large-scale LVLMs?
• RQ2 Does using GPT-4V to generate complex, detailed instructions and answers improve reasoning and alignment in small vision-language models?
• RQ3 How effective is a two-stage training pipeline—pretraining on alignment data followed by instruction tuning—when using synthetic data?
• RQ4 Can a 3B-parameter model achieve performance comparable to 13B–34B models when trained on high-quality synthetic data?

Key findings

• ALLaVA achieved a MME score of 1623.2 and a GQA score of 50.0, outperforming LLaVA-v1.5-13B on both benchmarks.
• The model achieved a MM-Vet score of 33.2, demonstrating strong multimodal reasoning capabilities despite its small size.
• Ablation studies showed that using both ALLaVA-Caption-4V and ALLaVA-Instruct-4V together yielded the best performance, with a 32.2 MM-Vet score when both were used.
• The Phi2-2.7B backbone outperformed larger LLMs like Qwen-1.8B and StableLM-2-1.6B across all benchmarks, indicating strong compatibility with the synthetic data.
• ALLaVA matched or exceeded GPT-4V’s reasoning quality on two qualitative examples, particularly in emotion recognition and humor comprehension.
• The open-sourced dataset of 1.4M samples is the largest GPT-4V-synthesized dataset for LVLM training to date, enabling future research in data-centric multimodal learning.