[Paper Review] Compositional Chain-of-Thought Prompting for Large Multimodal Models

Chancharik Mitra, Brandon Huang+2 more|arXiv (Cornell University)|Nov 27, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes Compositional Chain-of-Thought (CCoT), a zero-shot prompting method that enhances Large Multimodal Models (LMMs) by generating scene graphs from images and tasks to improve compositional visual reasoning. Without fine-tuning or ground-truth scene graph annotations, CCoT boosts performance on compositional benchmarks like Winoground and WHOOPS! and general multimodal benchmarks such as MMBench and SEEDBench.

ABSTRACT

The combination of strong visual backbones and Large Language Model (LLM) reasoning has led to Large Multimodal Models (LMMs) becoming the current standard for a wide range of vision and language (VL) tasks. However, recent research has shown that even the most advanced LMMs still struggle to capture aspects of compositional visual reasoning, such as attributes and relationships between objects. One solution is to utilize scene graphs (SGs)--a formalization of objects and their relations and attributes that has been extensively used as a bridge between the visual and textual domains. Yet, scene graph data requires scene graph annotations, which are expensive to collect and thus not easily scalable. Moreover, finetuning an LMM based on SG data can lead to catastrophic forgetting of the pretraining objective. To overcome this, inspired by chain-of-thought methods, we propose Compositional Chain-of-Thought (CCoT), a novel zero-shot Chain-of-Thought prompting method that utilizes SG representations in order to extract compositional knowledge from an LMM. Specifically, we first generate an SG using the LMM, and then use that SG in the prompt to produce a response. Through extensive experiments, we find that the proposed CCoT approach not only improves LMM performance on several vision and language VL compositional benchmarks but also improves the performance of several popular LMMs on general multimodal benchmarks, without the need for fine-tuning or annotated ground-truth SGs. Code: https://github.com/chancharikmitra/CCoT

Motivation & Objective

• To improve compositional visual reasoning in Large Multimodal Models (LMMs), particularly in understanding object attributes and relationships.
• To address the limitations of LMMs that treat images as 'bags of objects' rather than structured scenes.
• To develop a prompting method that leverages scene graphs without requiring annotated scene graph datasets or model fine-tuning.
• To enhance LMM performance on both compositional and general multimodal benchmarks using a token-efficient, generalizable approach.
• To enable zero-shot, scalable, and architecture-agnostic prompting that extracts deeper compositional knowledge from pre-trained LMMs.

Proposed method

• First, the LMM generates a scene graph from the input image and task prompt (e.g., a question), using its internal reasoning to extract objects, attributes, and relationships.
• Second, the LMM is prompted again with the image, the generated scene graph, and the original task prompt to produce a final answer.
• The method uses a two-step prompting pipeline: scene graph generation followed by answer generation, both within a single LLM inference context.
• Scene graphs serve as an intermediate, structured representation that guides the LLM toward compositional reasoning by making relationships and attributes explicit.
• The approach avoids fine-tuning and prevents catastrophic forgetting by not updating model weights, relying solely on prompt engineering.
• The method is generalizable across different LMM architectures and tasks, including VQA, captioning, and reasoning, due to the compact and expressive nature of scene graphs.

Experimental results

Research questions

• RQ1 Can a zero-shot prompting method that uses scene graphs improve compositional reasoning in LMMs without fine-tuning?
• RQ2 How effective is scene graph generation via LLMs in enhancing LMM performance on compositional visual reasoning benchmarks?
• RQ3 Does the use of generated scene graphs lead to performance gains on general multimodal benchmarks like MMBench and SEEDBench?
• RQ4 Can this method be applied across diverse LMM architectures without task-specific adaptation?
• RQ5 Does the CCoT approach mitigate the 'bag-of-objects' bias in LMMs by making relational and attribute-level reasoning more explicit?

Key findings

• CCoT significantly improves LLaVA-1.5 and InstructBLIP performance on the Winoground compositional reasoning benchmark, demonstrating enhanced understanding of object relationships and attributes.
• The method achieves state-of-the-art zero-shot performance on the WHOOPS! VQA benchmark, indicating strong generalization to out-of-distribution, commonsense-challenging images.
• CCoT improves performance on general multimodal benchmarks such as MMBench and SEEDBench, showing broader applicability beyond compositional reasoning tasks.
• The approach achieves these gains without any fine-tuning, model updates, or reliance on annotated scene graph datasets, making it scalable and efficient.
• Qualitative analysis shows that CCoT generates more accurate and reasoning-rich responses, especially in cases requiring relational understanding.
• The method is robust across different LMM architectures and maintains performance across diverse visual and language tasks, confirming its generalizability.