[Paper Review] MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressions

Henghui Ding, Chang Liu+3 more|arXiv (Cornell University)|Aug 16, 2023

Multimodal Machine Learning ApplicationsComputer Science3 citations

TL;DR

This paper introduces MeViS, a large-scale benchmark for video segmentation using motion expressions, emphasizing temporal motion over static attributes. It proposes a novel baseline method, LMPM, which uses language-conditional queries and motion-aware feature aggregation to improve segmentation performance, achieving a 37.2% J&F score—significantly higher than prior methods—highlighting the challenge of motion-guided segmentation in complex scenes.

ABSTRACT

This paper strives for motion expressions guided video segmentation, which focuses on segmenting objects in video content based on a sentence describing the motion of the objects. Existing referring video object datasets typically focus on salient objects and use language expressions that contain excessive static attributes that could potentially enable the target object to be identified in a single frame. These datasets downplay the importance of motion in video content for language-guided video object segmentation. To investigate the feasibility of using motion expressions to ground and segment objects in videos, we propose a large-scale dataset called MeViS, which contains numerous motion expressions to indicate target objects in complex environments. We benchmarked 5 existing referring video object segmentation (RVOS) methods and conducted a comprehensive comparison on the MeViS dataset. The results show that current RVOS methods cannot effectively address motion expression-guided video segmentation. We further analyze the challenges and propose a baseline approach for the proposed MeViS dataset. The goal of our benchmark is to provide a platform that enables the development of effective language-guided video segmentation algorithms that leverage motion expressions as a primary cue for object segmentation in complex video scenes. The proposed MeViS dataset has been released at https://henghuiding.github.io/MeViS.

Motivation & Objective

• To address the lack of benchmarks that emphasize motion expressions in video segmentation, moving beyond static attributes.
• To investigate the feasibility of using motion as the primary cue for grounding and segmenting objects in complex video scenes.
• To challenge existing referring video object segmentation (RVOS) methods by introducing a dataset with high temporal complexity and multi-object motion expressions.
• To provide a foundation for developing more robust, motion-aware language-guided video segmentation models.
• To enable research into global temporal modeling, multi-object tracking, and cross-modal understanding in dynamic video environments.

Proposed method

• Proposes a language-conditional query generation mechanism to detect potential target objects using object embeddings, enhancing robustness and efficiency.
• Applies motion perception on object embeddings to capture long-term and fleeting motion patterns across the entire video sequence.
• Uses a Transformer decoder to decode language features and predict object trajectories based on motion-aggregated embeddings.
• Introduces a similarity matching module between language features and predicted trajectories to identify target objects, supporting both single- and multi-object expressions.
• Employs a two-stage training strategy: first, motion perception on object embeddings; second, trajectory prediction and matching with language signals.
• Designs a baseline model, LMPM (Language-guided Motion Perception and Matching), to demonstrate the effectiveness of motion-aware modeling in complex video understanding.

Experimental results

Research questions

• RQ1 Can existing RVOS methods effectively segment objects based on motion expressions alone, without relying on static visual attributes?
• RQ2 How does the performance of state-of-the-art RVOS models degrade when motion is the primary cue rather than static appearance?
• RQ3 What are the key challenges in modeling long-term and fleeting motions across complex video sequences using language expressions?
• RQ4 Can a unified model effectively handle both single-object and multi-object motion expressions in a single framework?
• RQ5 What role does global temporal context play in accurately grounding motion expressions in videos?

Key findings

• Current state-of-the-art RVOS methods achieve only 27.8% to 31.0% J&F on MeViS, significantly lower than their 60%+ performance on existing benchmarks like Refer-Youtube-VOS and DAVIS 17-RVOS.
• The proposed LMPM baseline achieves a 37.2% J&F score on MeViS, outperforming all prior methods, demonstrating the need for motion-aware modeling.
• Failure cases reveal that models struggle with target object disappearance and reappearance, indicating limitations in global temporal reasoning.
• Complex motion interactions, such as tangled trajectories of multiple moving objects, lead to tracking failures, especially in later video frames.
• The multi-object expression capability of MeViS enables more realistic and challenging evaluation scenarios, which current models fail to handle robustly.
• The success of LMPM in handling long-term motion expressions like 'moving to the front' and 'going out of the screen' confirms the value of motion perception and trajectory prediction in the framework.