[Paper Review] Learning Motion Patterns in Videos

Pavel Tokmakov, Karteek Alahari+1 more|arXiv (Cornell University)|Dec 21, 2016

Advanced Vision and Imaging37 references4 citations

TL;DR

This paper proposes MP-Net, a fully convolutional network that learns motion patterns from synthetic video data to distinguish object motion from camera motion, achieving state-of-the-art performance in moving object segmentation. By leveraging optical flow, objectness maps, and CRF refinement, it outperforms prior methods by 5.6% on mean IoU in the DAVIS benchmark, even with only frame-level input.

ABSTRACT

The problem of determining whether an object is in motion, irrespective of camera motion, is far from being solved. We address this challenging task by learning motion patterns in videos. The core of our approach is a fully convolutional network, which is learned entirely from synthetic video sequences, and their ground-truth optical flow and motion segmentation. This encoder-decoder style architecture first learns a coarse representation of the optical flow field features, and then refines it iteratively to produce motion labels at the original high-resolution. We further improve this labeling with an objectness map and a conditional random field, to account for errors in optical flow, and also to focus on moving "things" rather than "stuff". The output label of each pixel denotes whether it has undergone independent motion, i.e., irrespective of camera motion. We demonstrate the benefits of this learning framework on the moving object segmentation task, where the goal is to segment all objects in motion. Our approach outperforms the top method on the recently released DAVIS benchmark dataset, comprising real-world sequences, by 5.6%. We also evaluate on the Berkeley motion segmentation database, achieving state-of-the-art results.

Motivation & Objective

• To address the long-standing challenge of distinguishing independent object motion from camera motion in videos.
• To develop a learning-based framework that automatically identifies moving objects regardless of camera motion.
• To improve moving object segmentation by combining motion pattern learning with objectness and CRF refinement.
• To achieve state-of-the-art performance on real-world video datasets without relying on video-level temporal propagation.
• To demonstrate generalization to diverse real-world sequences with complex motion patterns.

Proposed method

• Train a fully convolutional encoder-decoder network (MP-Net) end-to-end on synthetic video sequences from FlyingThings3D with ground-truth optical flow and motion segmentation.
• Use optical flow fields as input to MP-Net, which predicts per-pixel scores indicating independent motion.
• Refine MP-Net predictions using an objectness map to emphasize salient moving objects and suppress 'stuff' like water or smoke.
• Apply a conditional random field (CRF) to enforce spatial consistency and correct local errors in motion labeling.
• Leverage only two-frame optical flow inputs, avoiding complex video-level inference or tracking.
• Integrate MP-Net predictions into existing video segmentation frameworks (e.g., FST [27]) to boost performance.

Experimental results

Research questions

• RQ1 Can a deep learning model trained on synthetic data generalize to real-world video motion segmentation tasks?
• RQ2 How effectively can a fully convolutional network learn to separate object motion from camera motion using only optical flow?
• RQ3 To what extent do objectness maps and CRF refinement improve motion labeling accuracy in the presence of optical flow errors?
• RQ4 Can a frame-level method outperform video-level methods in moving object segmentation without temporal propagation?
• RQ5 How does the model perform on ambiguous or complex motion scenarios involving slow-moving or small objects?

Key findings

• MP-Net outperforms the top method on the DAVIS benchmark by 5.6% in mean intersection over union (IoU), achieving 72.1% IoU.
• The method improves the F-measure by 7% over the best prior method (NLC) on DAVIS, reaching 85.3% mean F-measure.
• When integrated into the FST [27] framework, MP-Net boosts performance by 14% on the BMS-26 dataset, matching the performance of PCM [3].
• On the FBMS test set, the combined MP+Obj+FST approach achieves 77.5% F-measure, outperforming FST (69.2), CVOS (74.9), and CUT (76.8).
• Qualitative results show superior boundary accuracy and reduced leakage into background regions compared to FST and NLC, especially on moving water and small objects.
• The method is robust to flow errors and effectively segments objects even when camera and object motion are similar, thanks to learned motion pattern discrimination.