[论文解读] What Makes for Good Visual Tokenizers for Large Language Models?

Guangzhi Wang, Yixiao Ge+3 more|arXiv (Cornell University)|May 20, 2023

Multimodal Machine Learning Applications被引用 5

一句话总结

本文研究了多模态大语言模型（MLLMs）的视觉分词器，发现自监督模型在细粒度感知方面表现优异，而监督模型则更擅长保持语义。作者提出GVT，一种通过无掩码方式蒸馏CLIP图像块特征的视觉分词器，在VQA、图像字幕生成、目标计数和多类别识别任务上实现了最先进性能，且无需增加参数量或进行任务特定微调。

ABSTRACT

We empirically investigate proper pre-training methods to build good visual tokenizers, making Large Language Models (LLMs) powerful Multimodal Large Language Models (MLLMs). In our benchmark, which is curated to evaluate MLLMs visual semantic understanding and fine-grained perception capabilities, we discussed different visual tokenizers pre-trained with dominant methods (i.e., DeiT, CLIP, MAE, DINO), and observe that: i) Fully/weakly supervised models capture more semantics than self-supervised models, but the gap is narrowed by scaling up the pre-training dataset. ii) Self-supervised models are better at fine-grained perception, where patch-level supervision is particularly effective. iii) Tuning the visual tokenizer leads to the loss of semantics obtained from large-scale pretraining, which is unfavorable with relatively small-scale instruction-tuning dataset. Given the findings, we reviewed methods that attempted to unify semantics and fine-grained visual understanding, e.g., patch-level feature distillation with semantically-rich targets. We obtain an intriguing insight mask-based strategies that were once all the rage may not be applicable for obtaining good visual tokenizers. Based on this critical observation, we obtain a new MLLM equipped with a tailored Good Visual Tokenizer (GVT), which exhibits strong visual comprehension capability at multiple scales. In particular, without introducing extra parameters and task-specific fine-tuning, GVT achieves superior performance on visual question answering, image captioning, and other fine-grained visual understanding tasks such as object counting and multi-class identification.

研究动机与目标

• 评估并比较在MLLM框架下用于语义理解与细粒度感知的视觉分词器。
• 识别现有预训练方法（尤其是基于掩码的策略）在指令微调过程中保持语义方面的局限性。
• 设计一种视觉分词器，统一强语义表征与高保真细粒度视觉感知能力。
• 开发一种方法，避免在端到端微调过程中出现训练-测试分布不匹配与语义退化。

提出的方法

• 构建了新的基准GVTBench，用于评估在语义（VQA、字幕生成）与细粒度（目标计数、多类别识别）任务中的视觉理解能力。
• 在相同MLLM架构上评估了通过DeiT（完全监督）、CLIP（弱监督）以及MAE/DINO/DINOv2（自监督）预训练的视觉分词器。
• 提出GVT，一种通过无掩码方式从CLIP图像块特征蒸馏训练的视觉分词器，以避免训练-测试分布不匹配。
• 在蒸馏过程中使用CLIP特征作为语义目标，以保留丰富语义信息，同时增强区域级理解能力。
• 在不引入额外参数或任务特定微调的前提下，将GVT与Vicuna端到端联合训练。
• 对比冻结与可微调视觉分词器，评估在小样本指令微调数据集上语义保持能力。

Figure 1: Different tasks require visual understanding of different perspectives. Mainstream vision-language tasks, e.g. , (a) VQA and (b) Image Captioning mainly focus on semantic understanding of the image. In this work, we also study two fine-grained visual understanding tasks: (c) Object Countin

实验结果

研究问题

• RQ1 不同的预训练范式（监督、弱监督、自监督）如何影响MLLM中视觉分词器的性能？
• RQ2 基于掩码的预训练策略（如MAE或MIM）能否在MLLM设置下的视觉分词器中有效应用，而不会引起分布偏移？
• RQ3 在小样本指令微调数据集上联合微调视觉分词器是否会导致语义退化，相较于冻结的视觉编码器？
• RQ4 能否设计一种统一的视觉分词器，同时保持强语义表征并支持细粒度视觉感知？

主要发现

• 完全监督和弱监督视觉分词器（如DeiT、CLIP）在语义理解方面优于自监督模型（如DINO、MAE），尽管在更大规模预训练数据下差距缩小。
• 自监督模型，尤其是具有图像块级别监督的模型，在目标计数和多类别识别等细粒度感知任务中表现更优。
• 在小样本指令微调数据集上联合微调视觉分词器会导致显著的语义退化，冻结的CLIP在语义任务上优于可微调的CLIP。
• 基于掩码的预训练策略（如MAE、MIM）会引入训练-测试分布不匹配，不适用于MLLM训练，因为其需要端到端优化，从而损害语义保持能力。
• 从CLIP图像块特征蒸馏到新模型（无掩码）成功保留了语义信息，同时增强了区域级理解能力。
• 所提出的GVT分词器在VQA、图像字幕生成、目标计数和多类别识别任务上实现了最先进性能，且无需额外参数或任务特定微调。

Figure 2: Framework of our GVT. We first distill the features of a pretrained CLIP via smoothed $\mathcal{L}_{1}$ loss. Then, we use it to encode images into a set of tokens, which are fed into the Perceiver Resampler [ 33 ] as soft prompts. Together with language instructions, these prompts are fed