[论文解读] InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning

Wenliang Dai, Junnan Li+7 more|arXiv (Cornell University)|May 11, 2023

Multimodal Machine Learning Applications被引用 401

一句话总结

InstructBLIP 将视觉—语言指令微调与冻结的 BLIP-2 主干以及一个面向指令的 Q-Former 相结合，在跨越多样化的视觉—语言任务时实现强大的零样本泛化，并且在下游微调中作为良好的初始化表现出色。

ABSTRACT

Large-scale pre-training and instruction tuning have been successful at creating general-purpose language models with broad competence. However, building general-purpose vision-language models is challenging due to the rich input distributions and task diversity resulting from the additional visual input. Although vision-language pretraining has been widely studied, vision-language instruction tuning remains under-explored. In this paper, we conduct a systematic and comprehensive study on vision-language instruction tuning based on the pretrained BLIP-2 models. We gather 26 publicly available datasets, covering a wide variety of tasks and capabilities, and transform them into instruction tuning format. Additionally, we introduce an instruction-aware Query Transformer, which extracts informative features tailored to the given instruction. Trained on 13 held-in datasets, InstructBLIP attains state-of-the-art zero-shot performance across all 13 held-out datasets, substantially outperforming BLIP-2 and larger Flamingo models. Our models also lead to state-of-the-art performance when finetuned on individual downstream tasks (e.g., 90.7% accuracy on ScienceQA questions with image contexts). Furthermore, we qualitatively demonstrate the advantages of InstructBLIP over concurrent multimodal models. All InstructBLIP models are open-sourced at https://github.com/salesforce/LAVIS/tree/main/projects/instructblip.

研究动机与目标

• 系统性地研究将大量公开数据集转化为指令格式的视觉—语言指令微调。
• 提出面向指令的视觉特征提取，以将视觉表征定制化为特定指令。
• 研究训练数据平衡及其对多数据集学习和零样本泛化的影响。
• 展示在多样化的视觉—语言任务和LLM骨干上的零样本和微调后性能。
• 发布开源模型并提供全面的基准测试与定性分析。

提出的方法

• 以预训练的 BLIP-2 模型（图像编码器、LLM 和 Q-Former）初始化。
• 仅对 Q-Former 进行微调，同时冻结图像编码器和 LLM 以实现指令微调。
• 将 26 个数据集转换为涵盖 11 个任务类别的指令模板，面向 13 个 held-in 和 13 个 held-out 数据集。
• 引入面向指令的 Q-Former，使视觉特征提取根据给定指令进行条件化。
• 应用与数据集大小平方根成比例的平衡采样策略，以稳定多数据集训练。
• 在推理阶段对分类和多项选择任务使用词汇排名生成方法。

Figure 1: A few qualitative examples generated by our InstructBLIP Vicuna model. Here, a range of its diverse capabilities are demonstrated, including complex visual scene understanding and reasoning, knowledge-grounded image description, multi-turn visual conversation, etc.

实验结果

研究问题

• RQ1 在统一模型平台下，视觉—语言指令微调是否能够对未见数据集和任务实现泛化？
• RQ2 面向指令的视觉特征提取是否提高了跨多样任务的任务相关性和零样本性能？
• RQ3 平衡多数据集训练对稳定性和总体泛化有哪些影响？
• RQ4 在不同的 LLM 骨干下，InstructBLIP 模型在零样本和下游微调场景中的表现如何？
• RQ5 在视觉—语言模型中，指令微调与多任务学习在零样本泛化方面的比较如何？

主要发现

• InstructBLIP 在 13 个 held-out 数据集的多样化视觉—语言任务中实现了新的零样本 SOTA。
• 在零样本设置中，指令微调始终超越 BLIP-2 和大型 Flamingo 模型。
• 面向指令的视觉特征带来显著提升，尤其在时空推理任务中。
• 平衡数据集采样提高训练稳定性和在 held-in 与 held-out 评估中的总体性能。
• 使用 InstructBLIP 进行微调更有效地初始化下游模型，在若干数据集上达到最先进的结果。

Figure 2: Tasks and their corresponding datasets used for vision-language instruction tuning. The held-in datasets are indicated by yellow and the held-out datasets by white.