[Paper Review] Learning to Segment Moving Objects in Videos

Katerina Fragkiadaki, Pablo Arbeláez+2 more|arXiv (Cornell University)|Dec 19, 2014

Video Surveillance and Tracking Methods33 references4 citations

TL;DR

This paper proposes a novel method for video moving object segmentation by generating per-frame motion-based object proposals (MOPs) from optical flow boundaries and ranking them using a dual-pathway CNN-based Moving Objectness Detector (MOD). The approach extends top-ranked proposals into spatio-temporal tubes via random walks on trajectory affinities, achieving state-of-the-art performance on VSB100 and Moseg benchmarks, with a 7% improvement in detection rate over static proposal methods.

ABSTRACT

We segment moving objects in videos by ranking spatio-temporal segment proposals according to "moving objectness": how likely they are to contain a moving object. In each video frame, we compute segment proposals using multiple figure-ground segmentations on per frame motion boundaries. We rank them with a Moving Objectness Detector trained on image and motion fields to detect moving objects and discard over/under segmentations or background parts of the scene. We extend the top ranked segments into spatio-temporal tubes using random walkers on motion affinities of dense point trajectories. Our final tube ranking consistently outperforms previous segmentation methods in the two largest video segmentation benchmarks currently available, for any number of proposals. Further, our per frame moving object proposals increase the detection rate up to 7\% over previous state-of-the-art static proposal methods.

Motivation & Objective

• To improve video object segmentation by leveraging motion cues to generate more accurate and discriminative object proposals.
• To address the limitations of static boundary detectors in cluttered scenes by using optical flow boundaries directly for proposal generation.
• To develop a trainable, class-agnostic objectness detector that distinguishes moving objects from static backgrounds and over/under-segmentations.
• To bridge the gap between motion segmentation and tracking by combining proposal ranking with spatio-temporal tube formation using dense trajectories.
• To achieve high segmentation accuracy with a minimal number of proposals by ranking tubes based on learned moving objectness scores.

Proposed method

• Per-frame Moving Object Proposals (MOPs) are generated by applying multiple figure-ground segmentations to motion boundaries derived from optical flow magnitude.
• Motion boundaries are detected using a learned boundary detector on optical flow, avoiding the need to combine with static boundaries to reduce bleeding artifacts.
• A dual-pathway Convolutional Neural Network (MOD) is trained on image and optical flow fields to predict a 'moving objectness' score for each proposal, filtering out false positives.
• Top-ranked MOPs are extended into 3D spatio-temporal tubes using random walkers on motion affinities computed from dense point trajectories across frames.
• Trajectory clusters are mapped to pixel tubes via overlap with supervoxels, ensuring spatial consistency across time.
• Tube scores are aggregated as the sum of objectness scores over their lifespan to favor longer, more stable tubes.

Experimental results

Research questions

• RQ1 Can motion-based object proposals improve detection rates for moving objects compared to static proposal methods?
• RQ2 How effective is a learned moving objectness detector in filtering over-segmented, under-segmented, or background proposals?
• RQ3 To what extent can spatio-temporal tube extension via trajectory random walks improve segmentation accuracy?
• RQ4 Does using optical flow boundaries directly, without fusion with static boundaries, lead to better proposal diversity and robustness in cluttered scenes?
• RQ5 Can a class-agnostic, dual-pathway CNN effectively learn to detect moving objects across diverse categories without prior knowledge of object classes?

Key findings

• The proposed MOPs increase the detection rate by up to 7% over state-of-the-art static proposal methods, especially in cluttered scenes like VSB100.
• The dual-pathway CNN-based Moving Objectness Detector (MOD) outperforms hand-coded saliency and other multilayer objectness baselines in ranking proposals.
• On the VSB100 dataset, combining GOPs and MOPs yields a significant performance boost, particularly in scenes where static boundary detectors fail.
• The method achieves state-of-the-art results on both VSB100 and Moseg benchmarks, with consistent superiority across all numbers of proposals.
• The MOD's score aggregation over tube lifespans improves long-term tracking stability, favoring longer, more coherent tubes.
• Failure cases are primarily due to temporal fragmentation from large motion or occlusions, suggesting a need for future linking heuristics.