[Paper Review] FusionNet and AugmentedFlowNet: Selective Proxy Ground Truth for Training on Unlabeled Images

Osama Makansi, Eddy Ilg+1 more|arXiv (Cornell University)|Aug 20, 2018

Advanced Neural Network Applications22 references3 citations

TL;DR

This paper proposes FusionNet, a method that combines multiple optical flow estimates using a learned assessment network to generate a high-quality proxy ground truth, and AugmentedFlowNet, a network fine-tuned on this proxy to enable unsupervised domain adaptation. The approach achieves state-of-the-art performance on KITTI benchmarks by leveraging unlabeled real-world videos to improve optical flow estimation accuracy and robustness.

ABSTRACT

Recent work has shown that convolutional neural networks (CNNs) can be used to estimate optical flow with high quality and fast runtime. This makes them preferable for real-world applications. However, such networks require very large training datasets. Engineering the training data is difficult and/or laborious. This paper shows how to augment a network trained on an existing synthetic dataset with large amounts of additional unlabelled data. In particular, we introduce a selection mechanism to assemble from multiple estimates a joint optical flow field, which outperforms that of all input methods. The latter can be used as proxy-ground-truth to train a network on real-world data and to adapt it to specific domains of interest. Our experimental results show that the performance of networks improves considerably, both, in cross-domain and in domain-specific scenarios. As a consequence, we obtain state-of-the-art results on the KITTI benchmarks.

Motivation & Objective

• To address the domain gap between synthetic training data and real-world images in optical flow estimation.
• To improve optical flow network performance on real-world data without requiring manual annotations.
• To enable domain-specific adaptation of optical flow networks using only unlabeled videos.
• To develop a proxy ground truth method that outperforms individual optical flow estimation techniques.
• To achieve state-of-the-art accuracy-runtime trade-offs in optical flow estimation through self-supervised adaptation.

Proposed method

• FusionNet is trained to predict the error of multiple input optical flow fields, enabling selection of the lowest-error flow vector per pixel.
• The selected flow fields from FusionNet serve as a high-quality proxy ground truth for fine-tuning a FlowNet.
• The resulting AugmentedFlowNet is trained on large-scale unlabeled video data, enabling domain adaptation without manual annotations.
• The method uses a multi-stage training pipeline: first, FusionNet is trained on synthetic data to assess flow quality; then, the proxy is used to fine-tune a FlowNet on real images.
• The assessment network leverages a loss function that minimizes the difference between predicted error and ground-truth error on synthetic datasets.
• The approach enables both cross-domain and domain-specific adaptation, with performance gains observed even without additional fine-tuning on ground truth.

Experimental results

Research questions

• RQ1 Can a learned ensemble of optical flow methods produce a proxy ground truth that exceeds the performance of any individual method?
• RQ2 Can a FlowNet be effectively fine-tuned using a proxy ground truth derived from multiple unsupervised flow estimators?
• RQ3 Does training on unlabeled real-world videos using a proxy ground truth improve optical flow accuracy on real benchmarks like KITTI?
• RQ4 How does the proposed method compare to fully unsupervised or synthetic-only training baselines in terms of domain adaptation performance?
• RQ5 Can the proxy ground truth approach generalize across domains and achieve state-of-the-art results without access to ground-truth labels?

Key findings

• FusionNet outperforms all individual input methods, producing a proxy ground truth that achieves 1.58 final flow error on the Sintel benchmark.
• The augmented FlowNet (AugmentedFlowNetG-CSS) achieves a final flow error of 2.10 on the KITTI 2012 benchmark, setting a new state of the art.
• After fine-tuning on KITTI's ground truth, the augmented FlowNet achieves 2.17% end-point error on KITTI 2015, surpassing previous SOTA results.
• The generic version of AugmentedFlowNet, trained only on the FusionNet proxy without additional ground truth, still achieves strong performance, outperforming baseline FlowNet models.
• Motion segmentation results on FBMS show a 4.77% improvement in F1 score when using augmented FlowNetC, demonstrating the benefit of real-world adaptation.
• The stacked augmented FlowNet even outperforms the FusionNet proxy in some cases, especially after fine-tuning with real ground truth, indicating the effectiveness of domain-specific adaptation.