[Paper Review] VIGC: Visual Instruction Generation and Correction

Bin Wang, Fan Wu+9 more|arXiv (Cornell University)|Aug 24, 2023

Multimodal Machine Learning ApplicationsComputer Science3 citations

TL;DR

This paper proposes VIGC, a framework that enables multimodal large language models to autonomously generate and iteratively correct high-quality vision-language instruction data, significantly reducing hallucinations. By combining Visual Instruction Generation (VIG) for diverse data creation and Visual Instruction Correction (VIC) with an iterative Q-Former update strategy, VIGC boosts downstream model performance, achieving state-of-the-art results on OKVQA and A-OKVQA with 1.6% improvement.

ABSTRACT

The integration of visual encoders and large language models (LLMs) has driven recent progress in multimodal large language models (MLLMs). However, the scarcity of high-quality instruction-tuning data for vision-language tasks remains a challenge. The current leading paradigm, such as LLaVA, relies on language-only GPT-4 to generate data, which requires pre-annotated image captions and detection bounding boxes, suffering from understanding image details. A practical solution to this problem would be to utilize the available multimodal large language models (MLLMs) to generate instruction data for vision-language tasks. However, it's worth noting that the currently accessible MLLMs are not as powerful as their LLM counterparts, as they tend to produce inadequate responses and generate false information. As a solution for addressing the current issue, this paper proposes the Visual Instruction Generation and Correction (VIGC) framework that enables multimodal large language models to generate instruction-tuning data and progressively enhance its quality on-the-fly. Specifically, Visual Instruction Generation (VIG) guides the vision-language model to generate diverse instruction-tuning data. To ensure generation quality, Visual Instruction Correction (VIC) adopts an iterative update mechanism to correct any inaccuracies in data produced by VIG, effectively reducing the risk of hallucination. Leveraging the diverse, high-quality data generated by VIGC, we finetune mainstream models and validate data quality based on various evaluations. Experimental results demonstrate that VIGC not only compensates for the shortcomings of language-only data generation methods, but also effectively enhances the benchmark performance. The models, datasets, and code are available at https://opendatalab.github.io/VIGC.

Motivation & Objective

• To address the scarcity of high-quality vision-language instruction tuning data, especially when relying on language-only models like GPT-4.
• To overcome the limitations of existing data generation methods that depend on costly human annotations and lose visual detail.
• To develop a self-supervised framework that enables multimodal models to generate and refine their own instruction data with reduced hallucination.
• To validate the effectiveness of VIGC-generated data through comprehensive evaluations on multiple benchmarks and model sizes.

Proposed method

• VIGC employs a two-stage framework: Visual Instruction Generation (VIG) to produce diverse image-question-answer triples using a vision-language model.
• Visual Instruction Correction (VIC) uses an iterative Q-Former (IQF) update mechanism to refine and correct hallucinated or inaccurate responses from VIG.
• The VIC module applies iterative refinement to improve answer precision by leveraging model self-consistency and knowledge grounding.
• The framework is trained on COCO and Objects365 datasets, generating 36,781 VIGC-LLaVA-COCO and ~1.8M VIGC-LLaVA-Objects365 samples.
• The generated data is used to fine-tune models like LLaVA and InstructBLIP, with performance evaluated on MMBench, LLaVA, OKVQA, and A-OKVQA.
• A closed-loop training strategy is explored, where self-generated data improves model performance, which in turn enhances future data quality.

Experimental results

Research questions

• RQ1 Can multimodal large language models generate diverse and high-quality vision-language instruction data without relying on language-only models?
• RQ2 Can iterative correction mechanisms effectively reduce hallucinations in self-generated multimodal instruction data?
• RQ3 Does data generated by VIGC outperform traditional instruction-tuning data in improving downstream model performance?
• RQ4 Can VIGC-generated data enhance model performance on zero-shot and few-shot benchmarks requiring external knowledge?
• RQ5 Can the self-improvement loop of data generation and model fine-tuning lead to sustained performance gains across multiple iterations?

Key findings

• VIGC-generated data, when used to fine-tune models, achieved 84.0%, 85.8%, and 83.3% performance on MMBench, LLaVA, and OKVQA, respectively, demonstrating strong generalization.
• The addition of VIC-corrected data led to a 1.6% improvement on A-OKVQA and 0.7% on OKVQA, achieving state-of-the-art results for models of this scale.
• Models fine-tuned on a mixture of LLaVA and VIGC-generated data outperformed models trained solely on LLaVA, even with constant data volume.
• Performance gains were observed across different model sizes, including LLaVA-13B, confirming scalability of the VIGC approach.
• Self-iterative training using VIGC-generated data improved model performance on both MMBench and LLaVA, indicating potential for closed-loop self-enhancement.
• The VIC module significantly reduced hallucinations, as evidenced by the marginal gain from raw VIG data versus substantial gains from VIC-refined data.