[Paper Review] CPSeg: Finer-grained Image Semantic Segmentation via Chain-of-Thought Language Prompting

Lei Li|arXiv (Cornell University)|Oct 24, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

CPSeg proposes a novel chain-of-thought language prompting framework for finer-grained semantic segmentation in remote sensing, particularly flood scenarios, by iteratively processing textual descriptions to enhance pixel-level classification. It achieves a 5.46% mIoU improvement over baseline methods on the FloodPrompt dataset, demonstrating superior accuracy and efficiency through structured, cognition-inspired reasoning with vision-language models.

ABSTRACT

Natural scene analysis and remote sensing imagery offer immense potential for advancements in large-scale language-guided context-aware data utilization. This potential is particularly significant for enhancing performance in downstream tasks such as object detection and segmentation with designed language prompting. In light of this, we introduce the CPSeg, Chain-of-Thought Language Prompting for Finer-grained Semantic Segmentation), an innovative framework designed to augment image segmentation performance by integrating a novel "Chain-of-Thought" process that harnesses textual information associated with images. This groundbreaking approach has been applied to a flood disaster scenario. CPSeg encodes prompt texts derived from various sentences to formulate a coherent chain-of-thought. We propose a new vision-language dataset, FloodPrompt, which includes images, semantic masks, and corresponding text information. This not only strengthens the semantic understanding of the scenario but also aids in the key task of semantic segmentation through an interplay of pixel and text matching maps. Our qualitative and quantitative analyses validate the effectiveness of CPSeg.

Motivation & Objective

• To address the challenge of fine-grained semantic segmentation in complex remote sensing images, particularly in flood disaster scenarios.
• To integrate human-like sequential reasoning into vision-language segmentation models by leveraging chain-of-thought prompting.
• To develop a new vision-language dataset, FloodPrompt, with detailed image-text annotations for training and evaluation.
• To improve segmentation performance by combining multi-sentence textual prompts with visual features through a structured reasoning process.
• To demonstrate the effectiveness of chain-of-thought prompting in enhancing both accuracy and computational efficiency in semantic segmentation.

Proposed method

• The framework employs a chain-of-thought process that decomposes image understanding into sequential, logical reasoning steps using multiple natural language prompts.
• Text prompts derived from diverse sentences are encoded using a text encoder and integrated with visual features via a cross-attention mechanism.
• A new vision-language dataset, FloodPrompt, is constructed with 1024×1024 remote sensing images, semantic masks, and corresponding descriptive prompts for fine-grained classes (e.g., flooded vs. non-flooded buildings and roads).
• The model uses a vision-language backbone (inspired by CLIP) and applies iterative prompting to refine segmentation predictions through progressive textual supervision.
• The method incorporates both explicit and implicit learning by combining multiple prompt types, including class-specific and spatial-contextual descriptions.
• Computational efficiency is maintained by optimizing FLOPs and parameters, achieving 42.85 FPS inference on A100 GPU.

Experimental results

Research questions

• RQ1 Can chain-of-thought language prompting improve the accuracy of fine-grained semantic segmentation in remote sensing imagery?
• RQ2 How does the integration of sequential, multi-sentence textual prompts affect segmentation performance compared to single-prompt or baseline methods?
• RQ3 To what extent can a vision-language model with a structured reasoning process generalize across different semantic classes in disaster scenarios?
• RQ4 Does the proposed framework maintain high efficiency while improving segmentation accuracy?
• RQ5 Can the model generalize to combined semantic labels (e.g., flooded and non-flooded classes) through implicit learning in the chain-of-thought process?

Key findings

• CPSeg achieves a mean Intersection over Union (mIoU) of 87.89 on the combined FloodPrompt dataset, representing a 5.46% improvement over the original data (82.43 mIoU).
• The model maintains high inference speed at 42.85 FPS on an NVIDIA A100 GPU, outperforming the baseline DenseCLIP in both FLOPs (1037.4 vs. 1043.1) and parameters (100.8G vs. 105.3G).
• Ablation studies confirm that both the chain-of-thought reasoning process and the text encoder are essential for performance gains, with ablation on either component leading to significant mIoU drops.
• The framework successfully generalizes to combined semantic labels (e.g., combining flooded and non-flooded classes), demonstrating robustness through implicit learning in the prompting chain.
• The method shows strong performance on fine-grained segmentation tasks, particularly in complex scenarios like flood disasters, where spatial and semantic context are critical.
• The results validate that structured, cognition-inspired reasoning via language prompting significantly enhances vision-language model performance in remote sensing segmentation.