[Paper Review] PG-Video-LLaVA: Pixel Grounding Large Video-Language Models

Shehan Munasinghe, R. Thushara+5 more|arXiv (Cornell University)|Nov 22, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

PG-Video-LLaVA is the first video-based large multimodal model with pixel-level grounding, integrating audio transcripts and a novel tracking-based grounding module to localize objects in response to natural language instructions. It achieves state-of-the-art performance on video grounding and zero-shot question-answering benchmarks using open-source Vicuna for reproducible evaluation.

ABSTRACT

Extending image-based Large Multimodal Models (LMMs) to videos is challenging due to the inherent complexity of video data. The recent approaches extending image-based LMMs to videos either lack the grounding capabilities (e.g., VideoChat, Video-ChatGPT, Video-LLaMA) or do not utilize the audio-signals for better video understanding (e.g., Video-ChatGPT). Addressing these gaps, we propose PG-Video-LLaVA, the first LMM with pixel-level grounding capability, integrating audio cues by transcribing them into text to enrich video-context understanding. Our framework uses an off-the-shelf tracker and a novel grounding module, enabling it to spatially localize objects in videos following user instructions. We evaluate PG-Video-LLaVA using video-based generative and question-answering benchmarks and introduce new benchmarks specifically designed to measure prompt-based object grounding performance in videos. Further, we propose the use of Vicuna over GPT-3.5, as utilized in Video-ChatGPT, for video-based conversation benchmarking, ensuring reproducibility of results which is a concern with the proprietary nature of GPT-3.5. Our framework builds on SoTA image-based LLaVA model and extends its advantages to the video domain, delivering promising gains on video-based conversation and grounding tasks. Project Page: https://github.com/mbzuai-oryx/Video-LLaVA

Motivation & Objective

• To address the lack of visual grounding in existing video-based LMMs, which limits precise object localization in responses.
• To enhance video understanding by incorporating audio signals through transcription, improving context in multimodal conversations.
• To enable reproducible evaluation of video conversational models by replacing proprietary GPT-3.5 with open-source Vicuna in benchmarking.
• To develop a modular, flexible framework for spatial grounding in videos that can be extended with future grounding techniques.
• To introduce new benchmarks specifically designed to evaluate prompt-based object grounding in video-language models.

Proposed method

• Extends the SoTA image-based LLaVA model to video by integrating spatio-temporal features from video clips with consistent camera views.
• Employs an off-the-shelf object tracker to maintain object localization across video frames, linking visual segments to conversational elements.
• Introduces a novel grounding module that maps textual referring expressions to pixel-level bounding boxes in video frames.
• Processes audio by transcribing it into text, which is then fused with visual and textual inputs to enrich video context understanding.
• Uses Vicuna-13b-v1.5 as the generative LLM for evaluation, ensuring transparency and reproducibility over proprietary models.
• Designs modular pipeline architecture to allow plug-and-play integration with future grounding or audio processing modules.

Experimental results

Research questions

• RQ1 Can a video-based LMM achieve accurate pixel-level grounding of objects in response to natural language instructions?
• RQ2 How does incorporating audio transcripts improve video understanding and grounding accuracy in multimodal conversations?
• RQ3 Can open-source LLMs like Vicuna replace proprietary models such as GPT-3.5 in benchmarking video conversational models without sacrificing reliability?
• RQ4 To what extent does the proposed grounding module outperform existing methods on video grounding and question-answering benchmarks?
• RQ5 How does the modular design of PG-Video-LLaVA support adaptability and integration with future advancements in visual grounding?

Key findings

• PG-Video-LLaVA achieves state-of-the-art performance on zero-shot video question-answering benchmarks, outperforming Video-ChatGPT, Video-LLaMA, FrozenBiLM, and Video Chat.
• On the VidSTG and HC-STVG benchmarks, PG-Video-LLaVA demonstrates superior spatial grounding accuracy, with qualitative results showing precise mask overlays on referred objects.
• Incorporating audio transcripts significantly improves understanding in complex scenes, as shown in Figure 5, where models without audio fail to capture key details.
• The use of Vicuna-13b-v1.5 for evaluation ensures reproducibility and transparency, addressing limitations of proprietary models like GPT-3.5-Turbo.
• The modular design enables seamless integration with other grounding modules and supports future enhancements in visual grounding technology.
• The model successfully localizes objects such as a tennis racket and a man in response to natural language prompts, as validated in qualitative examples (Figure 1).