[Paper Review] CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputs

Daoan Zhang, Junming Yang+5 more|arXiv (Cornell University)|Jan 5, 2024

Multimodal Machine Learning Applications4 citations

TL;DR

CoCoT is a novel contrastive chain-of-thought prompting method that enhances large multimodal models' understanding of multiple images by explicitly guiding them to compare similarities and differences before answering. It significantly improves performance on multi-image reasoning tasks, achieving state-of-the-art results across both open-source and closed-source models like GPT-4V and MMICL.

ABSTRACT

When exploring the development of Artificial General Intelligence (AGI), a critical task for these models involves interpreting and processing information from multiple image inputs. However, Large Multimodal Models (LMMs) encounter two issues in such scenarios: (1) a lack of fine-grained perception, and (2) a tendency to blend information across multiple images. We first extensively investigate the capability of LMMs to perceive fine-grained visual details when dealing with multiple input images. The research focuses on two aspects: first, image-to-image matching (to evaluate whether LMMs can effectively reason and pair relevant images), and second, multi-image-to-text matching (to assess whether LMMs can accurately capture and summarize detailed image information). We conduct evaluations on a range of both open-source and closed-source large models, including GPT-4V, Gemini, OpenFlamingo, and MMICL. To enhance model performance, we further develop a Contrastive Chain-of-Thought (CoCoT) prompting approach based on multi-input multimodal models. This method requires LMMs to compare the similarities and differences among multiple image inputs, and then guide the models to answer detailed questions about multi-image inputs based on the identified similarities and differences. Our experimental results showcase CoCoT's proficiency in enhancing the multi-image comprehension capabilities of large multimodal models.

Motivation & Objective

• To investigate the limitations of large multimodal models (LMMs) in fine-grained perception and information blending across multiple images.
• To address the lack of effective prompting strategies that guide LMMs to reason jointly across multiple inputs.
• To develop a prompting method that enhances LMMs’ ability to extract and summarize detailed visual relationships from multiple images.
• To evaluate the effectiveness of the proposed method across diverse open-source and closed-source LMMs.

Proposed method

• CoCoT introduces a two-stage prompting process: first, the model identifies similarities and differences across multiple input images.
• The method uses contrastive reasoning to highlight key visual distinctions, reducing information blending and improving focus on relevant details.
• It integrates chain-of-thought reasoning with visual contrast to guide step-by-step analysis of image pairs.
• The prompting framework is applied without fine-tuning, enabling direct deployment on existing LMMs.
• It is designed to be modular and compatible with various LMM architectures, including GPT-4V, Gemini, OpenFlamingo, and MMICL.
• The method is evaluated through ablation studies to isolate the contributions of similarity and difference prompting components.

Experimental results

Research questions

• RQ1 Can existing large multimodal models accurately match images based on fine-grained visual details when multiple inputs are present?
• RQ2 How does contrastive reasoning between multiple images improve multi-image comprehension compared to standard chain-of-thought prompting?
• RQ3 To what extent does CoCoT reduce information blending and improve focus on relevant visual distinctions in multimodal reasoning?
• RQ4 Does the performance gain from CoCoT generalize across diverse open-source and closed-source LMMs?
• RQ5 What is the relative contribution of identifying similarities versus differences in the CoCoT prompting framework?

Key findings

• CoCoT significantly improves multi-image-to-text matching accuracy, achieving 77.00% on MMICL and 77.80% on Gemini, outperforming baseline methods.
• On GPT-4V, CoCoT boosts the group score from 37.75% to 44.50%, indicating stronger alignment between image and text understanding.
• The ablation study shows that combining similarity and difference prompts yields the best performance, with MMICL reaching 77.00% when both components are used.
• Gemini’s performance improves from 25.00% to 27.75% with CoCoT, but remains below other models, suggesting limitations in summarizing detailed visual information.
• OpenFlamingo and MMICL show the most substantial gains with CoCoT, indicating strong compatibility and effectiveness on open-source models.
• Human performance remains significantly higher (85.50% group score), highlighting the gap between current models and human-level perception.