[Paper Review] Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language

Andy Zeng, Attarian, Maria+11 more|arXiv (Cornell University)|Apr 1, 2022

Multimodal Machine Learning ApplicationsComputer Science170 citations

TL;DR

The paper introduces Socratic Models (SMs), a modular zero-shot framework that composes pretrained multimodal models via language prompts to solve new multimodal tasks without finetuning, achieving competitive zero-shot results and enabling new applications.

ABSTRACT

Large pretrained (e.g., "foundation") models exhibit distinct capabilities depending on the domain of data they are trained on. While these domains are generic, they may only barely overlap. For example, visual-language models (VLMs) are trained on Internet-scale image captions, but large language models (LMs) are further trained on Internet-scale text with no images (e.g., spreadsheets, SAT questions, code). As a result, these models store different forms of commonsense knowledge across different domains. In this work, we show that this diversity is symbiotic, and can be leveraged through Socratic Models (SMs): a modular framework in which multiple pretrained models may be composed zero-shot i.e., via multimodal-informed prompting, to exchange information with each other and capture new multimodal capabilities, without requiring finetuning. With minimal engineering, SMs are not only competitive with state-of-the-art zero-shot image captioning and video-to-text retrieval, but also enable new applications such as (i) answering free-form questions about egocentric video, (ii) engaging in multimodal assistive dialogue with people (e.g., for cooking recipes) by interfacing with external APIs and databases (e.g., web search), and (iii) robot perception and planning.

Motivation & Objective

• Leverage complementary knowledge across pretrained models trained in different domains (vision, language, audio) without fine-tuning.
• Demonstrate that language can serve as a universal medium for inter-model communication to form multimodal inferences.
• Show competitive zero-shot performance on standard benchmarks and enable new applications like egocentric reasoning, multimodal dialogue, and robot planning.
• Highlight practical advantages, limitations, and directions for future work in assembling heterogeneous foundation models through prompting.

Proposed method

• Define Socratic Models as a computation graph of modules connected through language prompts.
• Use multimodal prompting to substitute non-language modality information into language prompts for LM-based reasoning.
• Employ a lightweight prompt-guided dialogue between VLMs, LMs, and ALMs (APIs) to perform zero-shot inference on downstream tasks.
• Demonstrate closed-loop interactions where the same models are called multiple times to refine outputs.
• Apply world-state history concepts by summarizing video content into language that LMs can reason about.

Experimental results

Research questions

• RQ1 Can zero-shot prompting across heterogeneous pretrained models (vision, language, audio, APIs) yield competitive multimodal task performance without training?
• RQ2 How can language serve as an intermediate representation to enable cross-modal exchange and reasoning among independent models?
• RQ3 What are the capabilities and limits of SMs for prompts-driven tasks like image captioning, contextual captioning, video retrieval, and egocentric reasoning?
• RQ4 Do SMs enable new applications such as multimodal assistive dialogue and robot perception/planning without task-specific finetuning?

Key findings

• SMs achieve competitive zero-shot performance on MS COCO image captioning and MSR-VTT video-to-text retrieval, with notable gains over some zero-shot baselines.
• On Concadia contextual captioning, SMs outperform a finetuned prior method in zero-shot captioning and description tasks.
• SMs set new zero-shot state-of-the-art in video-to-text retrieval on MSR-VTT for long-transcript videos, approaching finetuned methods in certain scenarios.
• SMs enable open-ended egocentric perception, multimodal assistive dialogue, and robot perception/planning by integrating external APIs and dynamic data sources.
• Prompts enable a self-contained, training-free framework that leverages the complementary strengths of VLMs, LMs, and ALMs, while acknowledging limitations such as reliance on pretrained model reliability.