[論文レビュー] InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning

Wenliang Dai, Junnan Li+7 more|arXiv (Cornell University)|May 11, 2023

Multimodal Machine Learning Applications被引用数 401

ひとこと要約

InstructBLIP は、視覚と言語の指示チューニングを凍結された BLIP-2 バックボーンと指示対応の Q-Former を組み合わせて、多様な視覚と言語タスクにおける強力なゼロショット汎化を実現し、下流のファインチューニングの良い初期化として優れている。

ABSTRACT

Large-scale pre-training and instruction tuning have been successful at creating general-purpose language models with broad competence. However, building general-purpose vision-language models is challenging due to the rich input distributions and task diversity resulting from the additional visual input. Although vision-language pretraining has been widely studied, vision-language instruction tuning remains under-explored. In this paper, we conduct a systematic and comprehensive study on vision-language instruction tuning based on the pretrained BLIP-2 models. We gather 26 publicly available datasets, covering a wide variety of tasks and capabilities, and transform them into instruction tuning format. Additionally, we introduce an instruction-aware Query Transformer, which extracts informative features tailored to the given instruction. Trained on 13 held-in datasets, InstructBLIP attains state-of-the-art zero-shot performance across all 13 held-out datasets, substantially outperforming BLIP-2 and larger Flamingo models. Our models also lead to state-of-the-art performance when finetuned on individual downstream tasks (e.g., 90.7% accuracy on ScienceQA questions with image contexts). Furthermore, we qualitatively demonstrate the advantages of InstructBLIP over concurrent multimodal models. All InstructBLIP models are open-sourced at https://github.com/salesforce/LAVIS/tree/main/projects/instructblip.

研究の動機と目的

• Systematically study vision-language instruction tuning using a wide range of public datasets transformed into instruction format.
• Propose instruction-aware visual feature extraction to tailor visual representations to specific instructions.
• Investigate training data balancing and its impact on multi-dataset learning and zero-shot generalization.
• Demonstrate zero-shot and finetuned performance across diverse vision-language tasks and LLM backbones.
• Release open-source models and provide comprehensive benchmarks and qualitative analyses.

提案手法

• Initialize with a pre-trained BLIP-2 model (image encoder, LLM, and Q-Former).
• Finetune only the Q-Former while keeping the image encoder and LLM frozen to enable instruction tuning.
• Transform 26 datasets into instruction templates spanning 11 task categories for 13 held-in and 13 held-out datasets.
• Introduce instruction-aware Q-Former that conditions visual feature extraction on the given instruction.
• Apply a balanced sampling strategy proportional to the square root of dataset sizes to stabilize multi-dataset training.
• Use a vocabulary ranking generation approach for classification and multi-choice tasks during inference.

Figure 1: A few qualitative examples generated by our InstructBLIP Vicuna model. Here, a range of its diverse capabilities are demonstrated, including complex visual scene understanding and reasoning, knowledge-grounded image description, multi-turn visual conversation, etc.

実験結果

リサーチクエスチョン

• RQ1 Can vision-language instruction tuning enable generalization to unseen datasets and tasks in a unified model platform?
• RQ2 Does instruction-aware visual feature extraction improve task relevance and zero-shot performance across diverse tasks?
• RQ3 What is the impact of balanced multi-dataset training on stability and overall generalization?
• RQ4 How do InstructBLIP models perform in zero-shot and downstream finetuning scenarios with different LLM backbones?
• RQ5 How does instruction tuning compare to multitask learning for zero-shot generalization in vision-language models?

主な発見

• InstructBLIP achieves new zero-shot SOTA on 13 held-out datasets across diverse vision-language tasks.
• Instruction tuning consistently outperforms BLIP-2 and large Flamingo models in zero-shot settings.
• Instruction-aware visual features provide substantial gains, particularly in spatial and temporal reasoning tasks.
• Balanced dataset sampling improves training stability and overall performance across held-in and held-out evaluations.
• Finetuning with InstructBLIP initializes downstream models more effectively, achieving state-of-the-art results on several datasets.

Figure 2: Tasks and their corresponding datasets used for vision-language instruction tuning. The held-in datasets are indicated by yellow and the held-out datasets by white.