[Paper Review] Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models

Chenfei Wu, Shengming Yin+4 more|arXiv (Cornell University)|Mar 8, 2023

Multimodal Machine Learning Applications187 citations

TL;DR

Visual ChatGPT integrates ChatGPT with multiple Visual Foundation Models via a Prompt Manager to handle image understanding, generation, and multi-step visual tasks through language. It enables iterative, multi-model visual reasoning without retraining a multi-modal agent.

ABSTRACT

ChatGPT is attracting a cross-field interest as it provides a language interface with remarkable conversational competency and reasoning capabilities across many domains. However, since ChatGPT is trained with languages, it is currently not capable of processing or generating images from the visual world. At the same time, Visual Foundation Models, such as Visual Transformers or Stable Diffusion, although showing great visual understanding and generation capabilities, they are only experts on specific tasks with one-round fixed inputs and outputs. To this end, We build a system called extbf{Visual ChatGPT}, incorporating different Visual Foundation Models, to enable the user to interact with ChatGPT by 1) sending and receiving not only languages but also images 2) providing complex visual questions or visual editing instructions that require the collaboration of multiple AI models with multi-steps. 3) providing feedback and asking for corrected results. We design a series of prompts to inject the visual model information into ChatGPT, considering models of multiple inputs/outputs and models that require visual feedback. Experiments show that Visual ChatGPT opens the door to investigating the visual roles of ChatGPT with the help of Visual Foundation Models. Our system is publicly available at \url{https://github.com/microsoft/visual-chatgpt}.

Motivation & Objective

• Extend ChatGPT with visual understanding and generation by leveraging existing Visual Foundation Models (VFMs) through a programmable Prompt Manager.
• Enable multi-turn, multi-model workflows where ChatGPT dispatches VFMs to perform complex visual tasks via natural language prompts.
• Improve reliability and coherence in multi-modal tasks by converting visual signals into language and managing model inputs/outputs and history.

Proposed method

• Design a Prompt Manager that (a) informs ChatGPT about VFMs and their input/output formats, (b) converts visual data to language, and (c) handles model histories, priorities, and conflicts.
• Introduce a pipeline where ChatGPT invokes multiple VFMs in sequence (e.g., depth estimation followed by depth-to-image followed by style transfer) to fulfill complex user requests.
• Define system principles and strict reasoning formats to prevent fabrications and to guide tool usage, including filename sensitivity and CoT-like multi-step reasoning.
• Represent intermediate VFM outputs as chained filenames to preserve provenance and facilitate subsequent steps.
• Support a broad set of 22 VFMs with inputs/outputs described via a structured prompt scheme, enabling zero-shot, multi-turn collaboration.

Experimental results

Research questions

• RQ1 How can a language model (ChatGPT) effectively orchestrate a diverse set of VFMs to perform multi-step visual tasks without retraining a multi-modal model?
• RQ2 What prompt engineering strategies (system prompts, VFM prompts, query prompts) enable reliable, interpretable, and extensible multi-model visual workflows?
• RQ3 Can a dispatcher-like Prompt Manager ensure correct VFM usage, data formatting, and provenance across complex image editing and generation tasks?

Key findings

• Visual ChatGPT enables language-based interaction with image understanding and generation through iterative VFM coordination.
• The Prompt Manager effectively maps non-language signals to language, defines VFM capabilities, and manages inputs/outputs and histories across models.
• Case studies and qualitative analyses show how system principles, VFM prompts, and query prompts influence success in multi-round, multi-model visual tasks.