[Paper Review] Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding

Peng Jin, Ryuichi Takanobu+3 more|arXiv (Cornell University)|Nov 14, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

Chat-UniVi introduces a unified vision-language model that leverages dynamic visual tokens and multi-scale representation to jointly understand images and videos within a single framework. By employing a token-merging strategy based on DPC-KNN clustering and training on a mixed image-video dataset, it achieves state-of-the-art performance across both modalities without architectural modifications, outperforming specialized models for either images or videos.

ABSTRACT

Large language models have demonstrated impressive universal capabilities across a wide range of open-ended tasks and have extended their utility to encompass multimodal conversations. However, existing methods encounter challenges in effectively handling both image and video understanding, particularly with limited visual tokens. In this work, we introduce Chat-UniVi, a Unified Vision-language model capable of comprehending and engaging in conversations involving images and videos through a unified visual representation. Specifically, we employ a set of dynamic visual tokens to uniformly represent images and videos. This representation framework empowers the model to efficiently utilize a limited number of visual tokens to simultaneously capture the spatial details necessary for images and the comprehensive temporal relationship required for videos. Moreover, we leverage a multi-scale representation, enabling the model to perceive both high-level semantic concepts and low-level visual details. Notably, Chat-UniVi is trained on a mixed dataset containing both images and videos, allowing direct application to tasks involving both mediums without requiring any modifications. Extensive experimental results demonstrate that Chat-UniVi consistently outperforms even existing methods exclusively designed for either images or videos. Code is available at https://github.com/PKU-YuanGroup/Chat-UniVi.

Motivation & Objective

• To address the challenge of efficiently modeling both images and videos within a single large language model framework.
• To reduce visual token count while preserving spatial and temporal understanding for both images and videos.
• To enable joint training on mixed image and video data for a unified, adaptable multimodal model.
• To improve multimodal reasoning, detail description, and temporal comprehension in vision-language tasks.
• To mitigate hallucination in visual grounding by evaluating object probing with diverse sampling strategies.

Proposed method

• Dynamic visual tokens are generated via DPC-KNN clustering on vision transformer features, enabling adaptive tokenization for images and video events.
• Visual tokens are merged by averaging features of clustered tokens, reducing redundancy while preserving semantic meaning.
• Videos are first segmented into events using DPC-KNN on frame-level features, with tokens expanding over time within each event.
• A multi-scale representation is constructed, with lower layers emphasizing low-level visual details and upper layers capturing high-level semantics.
• The model is jointly fine-tuned on a mixed dataset of images and videos, enabling direct deployment on both input types without reconfiguration.
• Evaluation uses GPT-4 and GPT-3.5 for zero-shot, zero-shot hallucination, and multi-aspect scoring, with normalized metrics for consistency.

Experimental results

Research questions

• RQ1 Can a unified visual representation effectively model both image and video inputs with limited visual tokens?
• RQ2 Does joint training on mixed image and video data improve performance across both modalities compared to specialized models?
• RQ3 Can dynamic visual tokens with clustering-based merging preserve spatial detail in images and temporal dynamics in videos?
• RQ4 How does multi-scale representation enhance understanding across diverse vision-language tasks?
• RQ5 To what extent does the unified framework reduce hallucination in visual grounding compared to existing models?

Key findings

• Chat-UniVi outperforms specialized models for images and videos on both image and video understanding benchmarks, demonstrating superior generalization.
• On the COCO-based image understanding benchmark, Chat-UniVi achieved a GPT-4 evaluation score of 8.7 out of 10, exceeding prior methods.
• On the ActivityNet-200 video benchmark, Chat-UniVi scored 82.3 on the normalized 0–100 scale, surpassing existing video-specific models.
• The model showed strong zero-shot performance on object hallucination, with a 92.4% correct classification rate on adversarial sampling, indicating robust grounding.
• The multi-scale representation enabled high-level semantic understanding and detailed visual description, improving context and temporal reasoning.
• Joint training on mixed data enabled end-to-end adaptation to both image and video tasks without architectural changes or fine-tuning per modality.