[Paper Review] Generalizable Entity Grounding via Assistance of Large Language Model

Qi Lu, Yiwen Chen+6 more|arXiv (Cornell University)|Feb 4, 2024

Data Quality and ManagementDecision Sciences3 citations

TL;DR

This paper proposes GELLA, a generalizable entity grounding framework that leverages a large language model (LMM) to extract semantic nouns from long captions and a class-agnostic segmentation model to generate entity masks. It introduces a colormap encoding strategy and a ResoBlend module to fuse low-resolution CLIP features with high-resolution mask features, enabling efficient, flexible, and accurate panoptic narrative grounding with significantly reduced computational cost, outperforming state-of-the-art methods on multiple benchmarks.

ABSTRACT

In this work, we propose a novel approach to densely ground visual entities from a long caption. We leverage a large multimodal model (LMM) to extract semantic nouns, a class-agnostic segmentation model to generate entity-level segmentation, and the proposed multi-modal feature fusion module to associate each semantic noun with its corresponding segmentation mask. Additionally, we introduce a strategy of encoding entity segmentation masks into a colormap, enabling the preservation of fine-grained predictions from features of high-resolution masks. This approach allows us to extract visual features from low-resolution images using the CLIP vision encoder in the LMM, which is more computationally efficient than existing approaches that use an additional encoder for high-resolution images. Our comprehensive experiments demonstrate the superiority of our method, outperforming state-of-the-art techniques on three tasks, including panoptic narrative grounding, referring expression segmentation, and panoptic segmentation.

Motivation & Objective

• To address the limitations of existing entity grounding methods that are rigid, computationally expensive, and tightly coupled to specific segmentation pipelines.
• To enable flexible, generalizable entity grounding from long captions by decoupling the captioning and segmentation components.
• To reduce computational cost by replacing high-resolution image encoders with a lightweight colormap encoder that preserves fine-grained mask features.
• To improve feature alignment between language and visual modalities using a ResoBlend module and CLIP-based embeddings.
• To design a framework that can integrate state-of-the-art segmentation models and LLMs without architectural constraints.

Proposed method

• Leverages a large multimodal model (LMM) to generate long captions and extract semantic nouns from them.
• Uses a class-agnostic segmentation model (e.g., EntitySeg) to produce entity-level segmentation masks.
• Encodes segmentation masks into a colormap with unique random colors to preserve high-resolution details while enabling low-cost processing.
• Employs a colormap encoder with a lightweight structure to extract visual features from the colormap, avoiding the need for high-resolution image encoders.
• Introduces the ResoBlend module to fuse features from the CLIP vision encoder (on low-res images) and the colormap encoder (on high-res masks) for improved feature consistency.
• Uses a learned association module to align $<$ SEG $>$ token embeddings with entity mask features, both derived from the CLIP image space, ensuring consistent cross-modal alignment.

Experimental results

Research questions

• RQ1 Can a framework achieve generalizable entity grounding from long captions without requiring high-resolution image encoders?
• RQ2 How effective is colormap-based mask encoding in preserving fine-grained segmentation details while reducing computational cost?
• RQ3 To what extent does the ResoBlend module improve feature fusion between low-res image features and high-res mask features?
• RQ4 Can the framework maintain high performance while being flexible enough to integrate diverse pre-trained LLMs and segmentation models?
• RQ5 How does the proposed method compare to SOTA in terms of accuracy and inference efficiency across multiple entity grounding tasks?

Key findings

• GELLA outperforms state-of-the-art methods on three benchmarks: panoptic narrative grounding (69.8 AR), referring expression segmentation, and panoptic segmentation.
• The colormap encoding strategy enables high-resolution mask feature preservation while reducing computational cost by over 85% compared to LISA, with inference time reduced from 5.5s to 1.2s.
• The ablation study shows that adding extra fully connected layers to the association module does not improve performance, confirming that CLIP-embedded features are already well-aligned in the same space.
• GELLA achieves 69.8 AR on the COCO panoptic narrative grounding benchmark using Mask2Former with a Swin-Large backbone, outperforming LISA (43.6 AR) and other SOTA methods.
• The framework demonstrates strong compatibility with various segmentation models, achieving 69.2 AR when using EntitySeg and 69.8 AR with Mask2Former, showing its generalization capability.
• The method maintains high performance even when using low-resolution image features from CLIP, proving that high-resolution image encoding is not necessary for accurate grounding.