[Paper Review] Learning Optical Flow via Dilated Networks and Occlusion Reasoning

Yi Zhu, Shawn Newsam|arXiv (Cornell University)|May 7, 2018

Advanced Vision and Imaging14 references3 citations

TL;DR

This paper proposes an unsupervised optical flow estimation method using dilated convolutions and occlusion reasoning to improve multi-scale handling and reduce gridding artifacts. By replacing deconvolution with dilated convolutions and incorporating dense skip connections, the model preserves high-resolution features and achieves state-of-the-art performance on KITTI benchmarks, with 2.35% F1-all error and 82.5% accuracy in action recognition.

ABSTRACT

Despite the significant progress that has been made on estimating optical flow recently, most estimation methods, including classical and deep learning approaches, still have difficulty with multi-scale estimation, real-time computation, and/or occlusion reasoning. In this paper, we introduce dilated convolution and occlusion reasoning into unsupervised optical flow estimation to address these issues. The dilated convolution allows our network to avoid upsampling via deconvolution and the resulting gridding artifacts. Dilated convolution also results in a smaller memory footprint which speeds up interference. The occlusion reasoning prevents our network from learning incorrect deformations due to occluded image regions during training. Our proposed method outperforms state-of-the-art unsupervised approaches on the KITTI benchmark. We also demonstrate its generalization capability by applying it to action recognition in video.

Motivation & Objective

• Address limitations in unsupervised optical flow estimation, including multi-scale handling, gridding artifacts, and occlusion reasoning.
• Reduce reliance on deconvolution to avoid resolution loss and gridding artifacts while maintaining high-resolution feature maps.
• Improve training stability and accuracy by explicitly modeling occlusions using forward-backward consistency.
• Enhance model generalization for downstream tasks like action recognition without requiring ground truth flow.
• Achieve real-time inference speed while maintaining high accuracy on real-world benchmarks.

Proposed method

• Replace deconvolutional upsampling with dilated convolutions in the final layers to preserve spatial resolution and avoid gridding artifacts.
• Apply dilated convolutions with dilation rates of 2 and 4 in later convolutional blocks (conv3_1, conv4_1, conv5_1) to expand receptive fields without downsampling.
• Integrate dense skip connections across layers to improve feature propagation and capture fine-grained motion, especially for small objects.
• Use an unsupervised training objective based on photometric consistency and brightness constancy, with gradients backpropagated only through non-occluded regions.
• Incorporate occlusion reasoning via forward-backward flow consistency: only update gradients where forward and backward flows are consistent.
• Train end-to-end using image reconstruction loss, with no need for ground truth optical flow.

Experimental results

Research questions

• RQ1 Can dilated convolutions improve multi-scale optical flow estimation without deconvolutional upsampling?
• RQ2 How does occlusion reasoning via forward-backward consistency improve unsupervised optical flow estimation?
• RQ3 To what extent does preserving high-resolution features through dilated convolutions and dense connections enhance flow accuracy?
• RQ4 Can an unsupervised optical flow model trained without ground truth achieve strong generalization to downstream tasks like action recognition?
• RQ5 Does the proposed method outperform existing unsupervised baselines on real-world benchmarks such as KITTI 2012 and 2015?

Key findings

• The proposed method achieves 2.35% F1-all error on the KITTI 2015 benchmark, outperforming state-of-the-art unsupervised methods.
• On KITTI 2012, the model reports 1.56% F1-all error, significantly improving over prior unsupervised approaches.
• The method achieves 82.5% top-1 accuracy on the UCF101 action recognition benchmark, surpassing FlowNet2 and FlowFields despite being unsupervised.
• The model runs at 33.3 fps, enabling real-time inference and outperforming FlowNet2 (8 fps) and FlowFields (0.06 fps).
• The use of dilated convolutions reduces memory footprint and eliminates gridding artifacts compared to deconvolution-based methods.
• The integration of occlusion reasoning leads to a 50%+ reduction in error rates on both occluded and non-occluded regions compared to non-occlusion-aware models.