[Paper Review] OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding

Tao Zhang, Xiangtai Li+6 more|arXiv (Cornell University)|Jun 27, 2024

Cell Image Analysis Techniques5 citations

TL;DR

OMG-LLaVA unifies image-level, object-level, and pixel-level reasoning in one model with one LLM, one visual encoder, and one visual decoder, enabling versatile multimodal tasks.

ABSTRACT

Current universal segmentation methods demonstrate strong capabilities in pixel-level image and video understanding. However, they lack reasoning abilities and cannot be controlled via text instructions. In contrast, large vision-language multimodal models exhibit powerful vision-based conversation and reasoning capabilities but lack pixel-level understanding and have difficulty accepting visual prompts for flexible user interaction. This paper proposes OMG-LLaVA, a new and elegant framework combining powerful pixel-level vision understanding with reasoning abilities. It can accept various visual and text prompts for flexible user interaction. Specifically, we use a universal segmentation method as the visual encoder, integrating image information, perception priors, and visual prompts into visual tokens provided to the LLM. The LLM is responsible for understanding the user's text instructions and providing text responses and pixel-level segmentation results based on the visual information. We propose perception prior embedding to better integrate perception priors with image features. OMG-LLaVA achieves image-level, object-level, and pixel-level reasoning and understanding in a single model, matching or surpassing the performance of specialized methods on multiple benchmarks. Rather than using LLM to connect each specialist, our work aims at end-to-end training on one encoder, one decoder, and one LLM. The code and model have been released for further research.

Motivation & Objective

• Motivate a single-model solution to bridge image-level, object-level, and pixel-level understanding and reasoning.
• Leverage a universal perception module to provide rich visual tokens to a frozen LLM.
• Achieve end-to-end task unification without multiple task-specific adapters or decoders.
• Preserve pixel-level segmentation ability while enabling image-level and region-level reasoning.
• Demonstrate strong performance across segmentation, grounding, and grounded conversation benchmarks.

Proposed method

• Use OMG-Seg as a frozen universal perception encoder to produce pixel-centric and object-centric visual tokens.
• Introduce a perception prior embedding module to fuse object queries with image features into token representations for the LLM.
• Encode image, region, and text prompts into a unified token format for the LLM to generate text and segmentation tokens.
• Decode [SEG] token outputs from the LLM into segmentation masks via the OMG decoder.
• Train in two stages: pretraining with frozen perception module and trainable projections; instruction tuning with LoRA for the LLM and supervision on segmentation losses.
• Adopt a unified instruction format with <Image>, <Region>, and [SEG] tokens to enable diverse tasks.

Experimental results

Research questions

• RQ1 Can image-, object-, and pixel-level tasks be solved within a single MLLM architecture with one encoder, one decoder, and one LLM?
• RQ2 Does perception prior embedding effectively integrate object-centric information into pixel features for LLM reasoning?
• RQ3 How does the unified approach perform on image captioning, VQA, referring segmentation, and grounded conversation tasks compared to specialized or multi-module systems?
• RQ4 What is the impact of visual prompts (points, boxes, masks) on object-level understanding and prompt-driven segmentation?
• RQ5 What data and training strategy are required to support broad multimodal capabilities without task-specific adapters?

Key findings

• OMG-LLaVA achieves image-, object-, and pixel-level understanding and reasoning within a single model.
• Perception prior embedding significantly improves segmentation-to-text alignment, boosting RES and GCG tasks.
• Providing object query inputs to the LLM yields additional performance gains on referring segmentation and grounded tasks.
• Compared with state-of-the-art MLLMs, OMG-LLaVA attains competitive or superior results on referring segmentation and grounded conversation while maintaining universal segmentation abilities.
• The model uses a single visual encoder and decoder, reducing complexity and computation relative to multi-encoder/decoder approaches.