[论文解读] Otter: A Multi-Modal Model with In-Context Instruction Tuning

Bo Li, Yuanhan Zhang+6 more|arXiv (Cornell University)|May 5, 2023

Topic Modeling被引用 87

一句话总结

Otter 对 OpenFlamingo 进行微调，在一个新的多模态上下文指令数据集上，以改进指令遵循和上下文学习，同时降低训练需求并与 Hugging Face 集成。

ABSTRACT

Recent advances in Large Multimodal Models (LMMs) have unveiled great potential as visual assistants. However, most existing works focus on responding to individual instructions or using previous dialogues for contextual understanding. There is little discussion on employing both images and text as in-context examples to enhance the instruction following capability. To bridge this gap, we introduce the extbf{Otter} model to leverage both textual and visual in-context examples for instruction tuning. Specifically, Otter builds upon Flamingo with Perceiver architecture, and has been instruction tuned for general purpose multi-modal assistant. Otter seamlessly processes multi-modal inputs, supporting modalities including text, multiple images, and dynamic video content. To support the training of Otter, we present the extbf{MIMIC-IT} ( extbf{M}ult extbf{I}- extbf{M}odal extbf{I}n- extbf{C}ontext extbf{I}nstruction extbf{T}uning) dataset, which encompasses over 3 million multi-modal instruction-response pairs, including approximately 2.2 million unique instructions across a broad spectrum of images and videos. MIMIC-IT has been carefully curated to feature a diverse array of in-context examples for each entry. Comprehensive evaluations suggest that instruction tuning with these in-context examples substantially enhances model convergence and generalization capabilities. Notably, the extensive scenario coverage provided by the MIMIC-IT dataset empowers the Otter model to excel in tasks involving complex video and multi-image understanding.

研究动机与目标

• 推动多模态模型的指令微调，以提高指令遵循和基于上下文的学习。
• 利用交错的多模态预训练数据，实现自然的跨模态对齐。
• 为研究人员提供实用、资源高效的微调工作流程。
• 通过将 Otter 与 Hugging Face 集成并降低硬件要求，实现获取的民主化。

提出的方法

• 引入包含带有上下文示例的图像-指令-答案三元组的 MIMIC-IT 数据集。
• 在冻结视觉编码器和语言解码器的同时，训练跨模态注意力和 Perceiver 重采样器模块，对 OpenFlamingo 基座进行微调，从而产生约 ~1.3B 的可训练参数。
• 使用带有特殊标记的聊天机器人风格训练格式来训练指令遵循和上下文学习。
• 使用 AdamW 在4 GPUs 上训练 6 轮，采用余弦学习率衰减和梯度裁剪。
• 将 Otter 集成到 Hugging Face Transformers，并提供 OpenFlamingo 检查点的转换脚本。

Figure 1 : Otter Overview . Otter is a multi-modal model finetuned on our proposed MIMIC-IT dataset, based on OpenFlamingo. Otter model exhibits the improved ability to execute tasks by following given instructions and leveraging in-context examples.

实验结果

研究问题

• RQ1 通过 MIMIC-IT 进行的多模态指令微调是否能提升多模态模型的显式指令遵循？
• RQ2 上下文学习是否使 Otter 能在少量示例的情况下执行新指令？
• RQ3 实现强大的多模态指令遵循所需的实际训练资源需求有哪些？
• RQ4 在指令遵循和场景理解方面，Otter 与 OpenFlamingo 的表现如何比较？
• RQ5 如何使 OpenFlamingo 架构对研究人员更加易于使用？

主要发现

• 在对 MIMIC-IT 进行微调后，Otter 展现出相较于 OpenFlamingo 的改进的指令遵循能力。
• Otter 能够利用提供的上下文示例学习执行新指令。
• 优化将训练需求降至 4× RTX3090 GPUs，并实现与 Hugging Face Transformers 的集成。
• 在定性分析中，该模型展现出比基线更深的场景理解和常识推理。
• Otter 以可访问的工具包发布，包括模型中心位置信息和转换脚本。

Figure 2 : Illustration of example data formats in MMC4 and MIMIC-IT . (a) The illustration of the data format in the MMC4 dataset that are used OpenFlamingo. (b) Three heuristics to build the multi-modal in-Context instruction tuning (MIMIC-IT) dataset.