[论文解读] PandaGPT: One Model To Instruction-Follow Them All

Yixuan Su, Lü Tian+4 more|arXiv (Cornell University)|May 25, 2023

Multimodal Machine Learning Applications被引用 46

一句话总结

PandaGPT 是一个多模态指令遵循模型，将 ImageBind 编码器与 Vicuna LLM 集成，以处理六种模态（image/video, text, audio, depth, thermal, IMU）使用对齐的 image-text 训练数据，使跨模态和组成任务成为可能。

ABSTRACT

We present PandaGPT, an approach to emPower large lANguage moDels with visual and Auditory instruction-following capabilities. Our pilot experiments show that PandaGPT can perform complex tasks such as detailed image description generation, writing stories inspired by videos, and answering questions about audios. More interestingly, PandaGPT can take multimodal inputs simultaneously and compose their semantics naturally. For example, PandaGPT can connect how objects look in an image/video and how they sound in an audio. To do so, PandaGPT combines the multimodal encoders from ImageBind and the large language models from Vicuna. Notably, only aligned image-text pairs are required for the training of PandaGPT. Thanks to the strong capability of ImageBind in embedding data from different modalities into the same space, PandaGPT displays emergent, i.e. zero-shot, cross-modal behaviors for data other than image and text (e.g., video, audio, depth, thermal, and IMU). We hope that PandaGPT serves as an initial step toward building AGI that can perceive and understand inputs in different modalities holistically, as we humans do. Our project page is at https://panda-gpt.github.io/.

研究动机与目标

• 推动构建一个能够感知并跨越多模态进行推理的通用模型。
• 利用对齐的 image-text 数据来训练一个多模态指令遵循系统。
• 探索由共享嵌入空间引发的跨模态新兴能力。
• 展示包括 image/video 描述、多模态叙事以及音频问答等任务。

提出的方法

• 将 ImageBind 多模态编码器与 Vicuna 13B 作为冻结的 LLM 主干结合。
• 仅训练一个线性投影 f 将 ImageBind 表示映射到 Vicuna，并在 Vicuna 注意力上使用 LoRA 权重。
• 使用 160k image-language instruction-following 例子，使用学习率 LR 5e-4，在 8x A100 GPU 上训练两轮。
• 在学习 Vicuna 参数的 0.4% 时保持 ImageBind 和 Vicuna 参数冻结。
• 目标在多模态提示与回应该序列上优化，表示为 pθ(y_i|x_<i,y_<i-1,f(h_I)).

Figure 1: Illustration of PandaGPT. During training, we only train the linear projection matrix and the additional LoRA weights (as indicated with dashed boxes) while keeping the parameters of ImageBind and Vicuna frozen.

实验结果

研究问题

• RQ1 PandaGPT 能否仅使用 image-text 对齐的训练数据，在六种模态上执行指令遵循？
• RQ2 将 ImageBind 嵌入与大型语言模型集成时，出现的跨模态能力有哪些？
• RQ3 PandaGPT 在视觉与听觉输入等模态之间在多大程度上能够组成语义？
• RQ4 哪些任务展示了多模态推理与组成性理解（如 image-video、image-audio）？

主要发现

• PandaGPT 支持基于 image/video 的问答、受 image/video 启发的写作，以及视觉与听觉推理。
• 它通过将来自不同模态的输入结合以产生综合输出，展示了多模态算术。
• 该模型在 image/video、text、audio、depth、thermal、IMU 上实现跨模态能力，无需模态特定训练。
• 尽管仅在对齐的 image-text 数据上进行训练，但由于 ImageBind 的共享嵌入空间，出现了新兴的跨模态行为。
• 训练设置使用了很小的模型参数比例（约 0.4% 的 Vicuna）。

Figure 2: Example showing PandaGPT’s capability in image-grounded question answering.