[Paper Review] u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model

Jinjin Xu, Liwu Xu+6 more|arXiv (Cornell University)|Nov 9, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

u-LLaVA proposes a unified multi-modal LLM framework that leverages a large language model as a cognitive bridge to connect task-specific expert models, reducing hallucinations and task interference. By reorganizing diverse public datasets—including a new 15K-salient-instruction dataset—it achieves state-of-the-art performance across vision tasks like grounding, segmentation, and captioning, with full code and model release.

ABSTRACT

Recent advancements in multi-modal large language models (MLLMs) have led to substantial improvements in visual understanding, primarily driven by sophisticated modality alignment strategies. However, predominant approaches prioritize global or regional comprehension, with less focus on fine-grained, pixel-level tasks. To address this gap, we introduce u-LLaVA, an innovative unifying multi-task framework that integrates pixel, regional, and global features to refine the perceptual faculties of MLLMs. We commence by leveraging an efficient modality alignment approach, harnessing both image and video datasets to bolster the model's foundational understanding across diverse visual contexts. Subsequently, a joint instruction tuning method with task-specific projectors and decoders for end-to-end downstream training is presented. Furthermore, this work contributes a novel mask-based multi-task dataset comprising 277K samples, crafted to challenge and assess the fine-grained perception capabilities of MLLMs. The overall framework is simple, effective, and achieves state-of-the-art performance across multiple benchmarks. We also make our model, data, and code publicly accessible at https://github.com/OPPOMKLab/u-LLaVA.

Motivation & Objective

• Address hallucinations and task interference in multi-modal LLMs (MLLMs) during downstream task adaptation.
• Unify diverse multi-modal tasks—such as visual grounding, segmentation, and captioning—under a single, scalable framework.
• Enable efficient, reproducible training by reorganizing and enhancing public datasets, especially for subjective tasks like salient segmentation.
• Leverage the world knowledge of LLMs to guide expert models without requiring end-to-end fine-tuning on ground-truth annotations.
• Achieve state-of-the-art performance across multiple benchmarks while maintaining simplicity and modularity.

Proposed method

• Integrate modality alignment and multi-task modules into the LLaVA architecture, using the LLM as a central reasoning hub.
• Reconstruct and reorganize public datasets—including a new Salient-15K dataset for salient segmentation—using BLIP2 and GPT to improve instruction quality and task specificity.
• Use the LLM to parse implicit label information from queries and guide expert models (e.g., GroundingDINO, SAM) for grounding and segmentation tasks.
• Implement a cross-verification workflow: generate bounding boxes via LLM parsing and validate them with mask2bbox outputs to improve robustness.
• Apply LoRA fine-tuning for efficient adaptation across tasks without full-parameter retraining.
• Decouple task-specific modules (e.g., grounding, segmentation) and connect them via the LLM’s reasoning capability to minimize interference.

Experimental results

Research questions

• RQ1 Can a unified MLLM framework reduce hallucinations and task interference by using the LLM as a reasoning bridge across expert models?
• RQ2 How does reorganizing and enhancing public datasets—especially for subjective tasks like salient segmentation—affect model generalization and performance?
• RQ3 To what extent can LLMs guide expert models (e.g., GroundingDINO, SAM) without direct supervision on bounding box or segmentation masks?
• RQ4 Can a simple, modular framework achieve state-of-the-art results across diverse multi-modal tasks including VQA, grounding, segmentation, and video captioning?
• RQ5 How does the cross-verification workflow between LLM-generated and mask2bbox-derived predictions improve grounding accuracy?

Key findings

• u-LLaVA achieves state-of-the-art performance on RefCOCOg test split with a Prec@0.5 of 72.52%, outperforming methods like Shikra that use ground-truth bounding boxes during training.
• The model achieves a 65.46% mAP on the DUT-OMRON salient object detection benchmark using the Salient-15K dataset, demonstrating improved understanding of subjective saliency.
• Ablation studies show that including all dataset types (referring, semantic, salient, captioning) in Stage II training yields the best generalization, with a 75.63% mAP on RefCOCOg under full fine-tuning.
• The cross-verification workflow between LLM-parsed and mask2bbox outputs improves grounding robustness, enabling high performance without end-to-end grounding training.
• The framework maintains strong performance across diverse tasks, including video captioning (67.78% on captioning benchmark) and inpainting, with qualitative results showing improved reasoning and generation quality.
• The open-sourcing of the model, code, and data—including the Salient-15K dataset—enables reproducibility and community extension of the framework.