[Paper Review] UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss

Simon Meister, Junhwa Hur+1 more|arXiv (Cornell University)|Nov 21, 2017

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes UnFlow, an unsupervised deep learning method for optical flow estimation that uses a bidirectional census loss to eliminate reliance on synthetic or real ground truth data. By combining occlusion-aware bidirectional flow estimation with a robust census transform, UnFlow achieves state-of-the-art performance on KITTI benchmarks, outperforming both prior unsupervised methods and supervised models trained on synthetic data alone.

ABSTRACT

In the era of end-to-end deep learning, many advances in computer vision are driven by large amounts of labeled data. In the optical flow setting, however, obtaining dense per-pixel ground truth for real scenes is difficult and thus such data is rare. Therefore, recent end-to-end convolutional networks for optical flow rely on synthetic datasets for supervision, but the domain mismatch between training and test scenarios continues to be a challenge. Inspired by classical energy-based optical flow methods, we design an unsupervised loss based on occlusion-aware bidirectional flow estimation and the robust census transform to circumvent the need for ground truth flow. On the KITTI benchmarks, our unsupervised approach outperforms previous unsupervised deep networks by a large margin, and is even more accurate than similar supervised methods trained on synthetic datasets alone. By optionally fine-tuning on the KITTI training data, our method achieves competitive optical flow accuracy on the KITTI 2012 and 2015 benchmarks, thus in addition enabling generic pre-training of supervised networks for datasets with limited amounts of ground truth.

Motivation & Objective

• To address the domain gap between synthetic training data and real-world test scenarios in optical flow estimation.
• To develop an unsupervised loss function that enables effective training on real-world image sequences without ground truth flow annotations.
• To improve robustness to occlusions and intensity variations in real images through bidirectional flow estimation and the census transform.
• To enable pre-training on large-scale real-world datasets, allowing fine-tuning with limited real annotations for state-of-the-art performance.
• To demonstrate that unsupervised training on real data can surpass supervised training on synthetic data in real-world benchmarks.

Proposed method

• Proposes a bidirectional optical flow estimation setup, where the network computes flow in both forward and backward directions using swapped image pairs.
• Introduces a novel unsupervised loss combining photometric reconstruction error with a census transform to enhance robustness to intensity variations.
• Incorporates occlusion reasoning by leveraging consistency between forward and backward flow predictions.
• Uses a higher-order smoothness term to regularize flow fields and reduce noise in motion estimates.
• Employs a multi-scale architecture to improve accuracy and generalization across different motion scales.
• Enables optional fine-tuning with sparse KITTI ground truth to further improve performance on challenging regions.

Experimental results

Research questions

• RQ1 Can an unsupervised loss based on bidirectional flow and census transformation outperform existing unsupervised optical flow methods on real-world benchmarks?
• RQ2 Does unsupervised training on real-world image sequences surpass supervised training on synthetic datasets in real-world performance?
• RQ3 Can the proposed loss enable effective pre-training on large-scale real datasets, even without ground truth, and support fine-tuning with limited annotations?
• RQ4 How does the inclusion of occlusion reasoning via bidirectional flow improve robustness in real-world scenes?
• RQ5 To what extent does the census transform enhance performance under intensity variations and real-world image distortions?

Key findings

• UnFlow-C, the purely unsupervised model trained on SYNTHIA and KITTI, outperforms all prior unsupervised deep networks on KITTI 2012 and 2015, reducing endpoint error by more than half compared to previous unsupervised methods.
• UnFlow-C achieves better accuracy than supervised FlowNetS and FlowNetC trained exclusively on synthetic data, demonstrating the advantage of real-world domain training.
• The fine-tuned UnFlow-CS-ft and UnFlow-CSS-ft models achieve performance competitive with FlowNet2-ft-kitti, a more complex supervised model, without requiring custom training schedules or auxiliary networks.
• UnFlow-C-Cityscapes, trained on Cityscapes without KITTI data, still outperforms UnsupFlownet and DSTFlow, showing strong generalization across domains.
• On the Middlebury benchmark, UnFlow-C and UnFlow-C-Cityscapes outperform the supervised FlowNetS, indicating strong transferability beyond the driving domain.
• On Sintel, UnFlow-C outperforms DSTFlow, showing that the method benefits even synthetic data it was not explicitly trained on, highlighting the robustness of the loss function.