[Paper Review] One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos

Zechen Bai, Tong He+7 more|arXiv (Cornell University)|Sep 29, 2024

Subtitles and Audiovisual Media4 citations

TL;DR

VideoLISA is a video-based multimodal large language model that enables language-instructed reasoning segmentation in videos by combining LLM reasoning with the Segment Anything Model. It introduces a Sparse Dense Sampling strategy for efficient spatiotemporal feature encoding and a One-Token-Seg-All approach using a <TRK> token to generate temporally consistent segmentation masks, achieving state-of-the-art performance on video and image segmentation benchmarks, including the new ReasonVOS benchmark.

ABSTRACT

We introduce VideoLISA, a video-based multimodal large language model designed to tackle the problem of language-instructed reasoning segmentation in videos. Leveraging the reasoning capabilities and world knowledge of large language models, and augmented by the Segment Anything Model, VideoLISA generates temporally consistent segmentation masks in videos based on language instructions. Existing image-based methods, such as LISA, struggle with video tasks due to the additional temporal dimension, which requires temporal dynamic understanding and consistent segmentation across frames. VideoLISA addresses these challenges by integrating a Sparse Dense Sampling strategy into the video-LLM, which balances temporal context and spatial detail within computational constraints. Additionally, we propose a One-Token-Seg-All approach using a specially designed token, enabling the model to segment and track objects across multiple frames. Extensive evaluations on diverse benchmarks, including our newly introduced ReasonVOS benchmark, demonstrate VideoLISA's superior performance in video object segmentation tasks involving complex reasoning, temporal understanding, and object tracking. While optimized for videos, VideoLISA also shows promising generalization to image segmentation, revealing its potential as a unified foundation model for language-instructed object segmentation. Code and model will be available at: https://github.com/showlab/VideoLISA.

Motivation & Objective

• To address the challenge of language-instructed reasoning segmentation in videos, which requires temporal understanding and consistent segmentation across frames.
• To overcome the limitations of image-based models like LISA when applied to video tasks due to the added temporal dimension.
• To enable a unified foundation model for both video and image segmentation using language instructions.
• To design a method that balances computational efficiency with high-fidelity spatiotemporal representation for dense prediction tasks.
• To evaluate the model’s reasoning, tracking, and generalization capabilities on a new benchmark, ReasonVOS.

Proposed method

• Introduces a Sparse Dense Sampling strategy that uniformly samples dense frames at full resolution and down-samples interleaved frames to reduce computational load while preserving temporal dynamics.
• Uses the inherent temporal redundancy in videos to balance spatial detail and temporal context in the input representation.
• Proposes a One-Token-Seg-All approach using a dedicated <TRK> token to encode unified spatiotemporal object representations across multiple frames.
• Trains the <TRK> token to predict segmentation masks for multiple frames simultaneously, preventing shortcut learning based on single-frame spatial cues.
• Integrates a multimodal LLM with the Segment Anything Model (SAM) to generate high-quality segmentation masks from natural language instructions.
• Employs a video-LLM architecture that processes visual features from sampled frames and uses the LLM to reason over both spatial and temporal content.

Experimental results

Research questions

• RQ1 Can a multimodal LLM effectively perform reasoning-based video object segmentation when given complex, natural language instructions?
• RQ2 How can temporal consistency in segmentation masks be achieved across video frames using a single prompt token?
• RQ3 What sampling strategy best balances computational efficiency with preservation of spatial and temporal information in video-LLM architectures?
• RQ4 To what extent can a video-LLM generalize to image segmentation tasks while maintaining strong reasoning and tracking capabilities?
• RQ5 How does the proposed One-Token-Seg-All approach compare to frame-by-frame or tracking-augmented baselines in terms of accuracy and consistency?

Key findings

• VideoLISA achieves state-of-the-art performance on multiple video object segmentation benchmarks, including motion-guided and referring segmentation tasks.
• The Sparse Dense Sampling strategy outperforms alternatives like frame concatenation, ST pooling, and slow-fast sampling, demonstrating its effectiveness in balancing detail and context.
• The One-Token-Seg-All approach significantly improves temporal consistency, outperforming both a baseline using a single <SEG> token and a LISA-based method augmented with an external tracker.
• Ablation studies confirm that the <TRK> token’s joint training across frames is essential for preventing shortcut learning and enabling cross-frame object association.
• VideoLISA generalizes well to image segmentation, showing strong performance on image reasoning and referring segmentation benchmarks.
• The model demonstrates superior reasoning and tracking capabilities on the newly introduced ReasonVOS benchmark, which evaluates complex, multi-frame reasoning tasks.