[Paper Review] Otter: A Multi-Modal Model with In-Context Instruction Tuning

Bo Li, Yuanhan Zhang+6 more|arXiv (Cornell University)|May 5, 2023

Topic ModelingComputer Science87 citations

TL;DR

Otter fine-tunes OpenFlamingo on a new multi-modal in-context instruction dataset to improve instruction following and in-context learning, while reducing training requirements and integrating with Hugging Face.

ABSTRACT

Recent advances in Large Multimodal Models (LMMs) have unveiled great potential as visual assistants. However, most existing works focus on responding to individual instructions or using previous dialogues for contextual understanding. There is little discussion on employing both images and text as in-context examples to enhance the instruction following capability. To bridge this gap, we introduce the extbf{Otter} model to leverage both textual and visual in-context examples for instruction tuning. Specifically, Otter builds upon Flamingo with Perceiver architecture, and has been instruction tuned for general purpose multi-modal assistant. Otter seamlessly processes multi-modal inputs, supporting modalities including text, multiple images, and dynamic video content. To support the training of Otter, we present the extbf{MIMIC-IT} ( extbf{M}ult extbf{I}- extbf{M}odal extbf{I}n- extbf{C}ontext extbf{I}nstruction extbf{T}uning) dataset, which encompasses over 3 million multi-modal instruction-response pairs, including approximately 2.2 million unique instructions across a broad spectrum of images and videos. MIMIC-IT has been carefully curated to feature a diverse array of in-context examples for each entry. Comprehensive evaluations suggest that instruction tuning with these in-context examples substantially enhances model convergence and generalization capabilities. Notably, the extensive scenario coverage provided by the MIMIC-IT dataset empowers the Otter model to excel in tasks involving complex video and multi-image understanding.

Motivation & Objective

• Motivate instruction tuning for multi-modal models to improve instruction following and context-based learning.
• Leverage interleaved multi-modal pretraining data to enable natural cross-modal alignment.
• Provide a practical, resource-efficient finetuning workflow for researchers.
• Democratize access by integrating Otter with Hugging Face and reducing hardware requirements.

Proposed method

• Introduce the MIMIC-IT dataset comprising image-instruction-answer triplets with contextual in-context examples.
• Finetune the OpenFlamingo base by freezing the vision encoder and language decoder while training cross-attention and Perceiver resampler modules, yielding ~1.3B trainable parameters.
• Use a chatbot-style training format with special tokens to train instruction following and in-context learning.
• Train with AdamW on 4 GPUs for 6 epochs using cosine learning rate decay and gradient clipping.
• Integrate Otter into Hugging Face Transformers and provide conversion scripts for OpenFlamingo checkpoints.

Figure 1 : Otter Overview . Otter is a multi-modal model finetuned on our proposed MIMIC-IT dataset, based on OpenFlamingo. Otter model exhibits the improved ability to execute tasks by following given instructions and leveraging in-context examples.

Experimental results

Research questions

• RQ1 Can multi-modal instruction tuning via MIMIC-IT improve explicit instruction following in a multi-modal model?
• RQ2 Does in-context learning enable Otter to execute new instructions with few exemplars?
• RQ3 What are the practical training resource requirements to achieve strong multi-modal instruction following?
• RQ4 How does Otter compare to OpenFlamingo in instruction following and scene understanding?
• RQ5 How can the OpenFlamingo architecture be made more accessible to researchers?

Key findings

• Otter demonstrates improved instruction-following ability over OpenFlamingo after fine-tuning on MIMIC-IT.
• Otter can learn to execute new instructions using provided in-context examples.
• Optimization reduces training requirements to 4× RTX3090 GPUs and enables integration into Hugging Face Transformers.
• The model showcases deeper scene understanding and commonsense reasoning compared with baselines in qualitative analyses.
• Otter is released with accessible tooling, including model hub placements and conversion scripts.

Figure 2 : Illustration of example data formats in MMC4 and MIMIC-IT . (a) The illustration of the data format in the MMC4 dataset that are used OpenFlamingo. (b) Three heuristics to build the multi-modal in-Context instruction tuning (MIMIC-IT) dataset.