[Paper Review] Deep Semantic Matching for Optical Flow.

Min Bai, Wenjie Luo+2 more|arXiv (Cornell University)|Apr 6, 2016

Advanced Vision and Imaging27 references3 citations

TL;DR

This paper proposes a deep learning approach for monocular optical flow estimation in autonomous driving by leveraging instance-level segmentation and epipolar constraints to model rigid motion of traffic participants. It introduces a novel convolutional network that performs semantic matching with uncertainty estimation, achieving state-of-the-art performance on the KITTI 2015 benchmark.

ABSTRACT

We tackle the problem of estimating optical flow from a monocular camera in the context of autonomous driving. We build on the observation that the scene is typically composed of a static background, as well as a relatively small number of traffic participants which move rigidly in 3D. We propose to estimate the traffic participants using instance-level segmentation. For each traffic participant, we use the epipolar constraints that govern each independent motion for faster and more accurate estimation. Our second contribution is a new convolutional net that learns to perform flow matching, and is able to estimate the uncertainty of its matches. This is a core element of our flow estimation pipeline. We demonstrate the effectiveness of our approach in the challenging KITTI 2015 flow benchmark, and show that our approach outperforms published approaches by a large margin.

Motivation & Objective

• To improve optical flow estimation in monocular autonomous driving scenarios by modeling traffic participants as rigidly moving entities.
• To reduce estimation error by exploiting epipolar geometry constraints specific to each moving object instance.
• To develop a deep network that learns to match features across frames while predicting uncertainty in matches.
• To enhance flow accuracy and robustness through instance-level segmentation and motion consistency.

Proposed method

• The method uses instance-level segmentation to isolate moving traffic participants from the static background.
• For each segmented object, epipolar constraints are applied to enforce geometric consistency in motion estimation.
• A novel convolutional neural network is trained to perform semantic feature matching between frames with uncertainty prediction.
• The network is integrated into a flow estimation pipeline that combines instance motion models with learned matching confidence.
• The approach leverages scene structure—static background and rigidly moving objects—to improve flow accuracy.
• Uncertainty-aware matching enables filtering of low-confidence correspondences, improving robustness.

Experimental results

Research questions

• RQ1 Can instance-level segmentation improve optical flow estimation in autonomous driving scenarios?
• RQ2 How can epipolar constraints be effectively integrated with deep learning for motion estimation?
• RQ3 Can a deep network learn to perform semantic matching while estimating uncertainty in correspondences?
• RQ4 Does uncertainty prediction enhance flow accuracy and robustness in complex driving scenes?
• RQ5 Can a hybrid approach combining geometric constraints and learned features outperform end-to-end flow networks?

Key findings

• The proposed method achieves state-of-the-art performance on the KITTI 2015 optical flow benchmark.
• The use of instance-level segmentation significantly improves flow accuracy for moving vehicles and pedestrians.
• Incorporating epipolar constraints leads to more geometrically consistent flow fields.
• The uncertainty estimation module effectively identifies and suppresses low-confidence matches.
• The model outperforms published approaches by a large margin, demonstrating superior generalization in real-world driving sequences.