[論文レビュー] Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models

Chenfei Wu, Shengming Yin+4 more|arXiv (Cornell University)|Mar 8, 2023

Multimodal Machine Learning Applications被引用数 187

ひとこと要約

Visual ChatGPT は ChatGPT を複数の Visual Foundation Models と統合し、Prompt Manager を介して画像理解、生成、そして言語を通じた多段の視覚タスクを処理します。これにより、多言語での反復的・多モデルの視覚推論を再訓練不要で可能にします。

ABSTRACT

ChatGPT is attracting a cross-field interest as it provides a language interface with remarkable conversational competency and reasoning capabilities across many domains. However, since ChatGPT is trained with languages, it is currently not capable of processing or generating images from the visual world. At the same time, Visual Foundation Models, such as Visual Transformers or Stable Diffusion, although showing great visual understanding and generation capabilities, they are only experts on specific tasks with one-round fixed inputs and outputs. To this end, We build a system called extbf{Visual ChatGPT}, incorporating different Visual Foundation Models, to enable the user to interact with ChatGPT by 1) sending and receiving not only languages but also images 2) providing complex visual questions or visual editing instructions that require the collaboration of multiple AI models with multi-steps. 3) providing feedback and asking for corrected results. We design a series of prompts to inject the visual model information into ChatGPT, considering models of multiple inputs/outputs and models that require visual feedback. Experiments show that Visual ChatGPT opens the door to investigating the visual roles of ChatGPT with the help of Visual Foundation Models. Our system is publicly available at \url{https://github.com/microsoft/visual-chatgpt}.

研究の動機と目的

• Extend ChatGPT with visual understanding and generation by leveraging existing Visual Foundation Models (VFMs) through a programmable Prompt Manager.
• Enable multi-turn, multi-model workflows where ChatGPT dispatches VFMs to perform complex visual tasks via natural language prompts.
• Improve reliability and coherence in multi-modal tasks by converting visual signals into language and managing model inputs/outputs and history.

提案手法

• Design a Prompt Manager that (a) informs ChatGPT about VFMs and their input/output formats, (b) converts visual data to language, and (c) handles model histories, priorities, and conflicts.
• Introduce a pipeline where ChatGPT invokes multiple VFMs in sequence (e.g., depth estimation followed by depth-to-image followed by style transfer) to fulfill complex user requests.
• Define system principles and strict reasoning formats to prevent fabrications and to guide tool usage, including filename sensitivity and CoT-like multi-step reasoning.
• Represent intermediate VFM outputs as chained filenames to preserve provenance and facilitate subsequent steps.
• Support a broad set of 22 VFMs with inputs/outputs described via a structured prompt scheme, enabling zero-shot, multi-turn collaboration.

実験結果

リサーチクエスチョン

• RQ1 How can a language model (ChatGPT) effectively orchestrate a diverse set of VFMs to perform multi-step visual tasks without retraining a multi-modal model?
• RQ2 What prompt engineering strategies (system prompts, VFM prompts, query prompts) enable reliable, interpretable, and extensible multi-model visual workflows?
• RQ3 Can a dispatcher-like Prompt Manager ensure correct VFM usage, data formatting, and provenance across complex image editing and generation tasks?

主な発見

• Visual ChatGPT enables language-based interaction with image understanding and generation through iterative VFM coordination.
• The Prompt Manager effectively maps non-language signals to language, defines VFM capabilities, and manages inputs/outputs and histories across models.
• Case studies and qualitative analyses show how system principles, VFM prompts, and query prompts influence success in multi-round, multi-model visual tasks.