[Paper Review] UPFlow: Upsampling Pyramid for Unsupervised Optical Flow Learning

Kunming Luo, Chuan Wang+4 more|arXiv (Cornell University)|Dec 1, 2020

Advanced Vision and Imaging45 references4 citations

TL;DR

This paper proposes UPFlow, an unsupervised optical flow method that improves pyramid-based networks through a self-guided upsampling module to reduce interpolation blur and a pyramid distillation loss that uses the finest flow as pseudo labels to supervise intermediate levels. The approach achieves state-of-the-art results, with EPE=1.4 on KITTI 2012 and F1=9.38% on KITTI 2015, outperforming prior unsupervised methods by 22.2% and 15.7% respectively.

ABSTRACT

We present an unsupervised learning approach for optical flow estimation by improving the upsampling and learning of pyramid network. We design a self-guided upsample module to tackle the interpolation blur problem caused by bilinear upsampling between pyramid levels. Moreover, we propose a pyramid distillation loss to add supervision for intermediate levels via distilling the finest flow as pseudo labels. By integrating these two components together, our method achieves the best performance for unsupervised optical flow learning on multiple leading benchmarks, including MPI-SIntel, KITTI 2012 and KITTI 2015. In particular, we achieve EPE=1.4 on KITTI 2012 and F1=9.38% on KITTI 2015, which outperform the previous state-of-the-art methods by 22.2% and 15.7%, respectively.

Motivation & Objective

• To address interpolation blur in pyramid-based optical flow networks caused by bilinear upsampling between levels.
• To improve supervision of intermediate pyramid levels in unsupervised optical flow learning, which are typically unguided.
• To enhance flow estimation accuracy by leveraging the finest flow as pseudo labels for distillation at coarser levels.
• To achieve state-of-the-art performance on unsupervised optical flow benchmarks without ground-truth labels.

Proposed method

• Introduces a self-guided upsampling module that learns an interpolation flow and map to guide upsampling, reducing blur on motion edges.
• Uses feature maps from the decoder as guidance to learn alignment-aware interpolation, improving detail preservation.
• Proposes a pyramid distillation loss that distills the finest flow as pseudo labels to supervise intermediate pyramid levels.
• Applies occlusion masks during distillation to exclude unreliable regions and improve pseudo-label quality.
• Combines the self-guided upsampling and pyramid distillation loss within a unified unsupervised training framework.
• Employs standard photometric, smoothness, and census losses alongside the new losses for end-to-end training.

Experimental results

Research questions

• RQ1 Can a learnable upsampling mechanism reduce interpolation blur in pyramid-based optical flow networks?
• RQ2 Does supervising intermediate pyramid levels with pseudo labels from the finest flow improve overall flow estimation?
• RQ3 Can a distillation-based loss strategy enhance performance in unsupervised optical flow learning without ground-truth supervision?
• RQ4 How does the integration of self-guided upsampling and pyramid distillation loss compare to prior unsupervised methods on standard benchmarks?

Key findings

• UPFlow achieves an EPE of 1.4 on the KITTI 2012 benchmark, representing a 22.2% relative improvement over the previous state-of-the-art.
• On the KITTI 2015 benchmark, UPFlow achieves an F1 score of 9.38%, a 15.7% relative improvement over the prior SOTA.
• The self-guided upsampling module outperforms image-guided methods like JBU, GF, and DJF, demonstrating superior edge preservation.
• Ablation studies show that removing the interpolation map or flow in the self-guided module degrades performance, confirming their importance.
• The pyramid distillation loss with occlusion masking yields better results than unsupervised losses (PUL-up/PUL-down), proving the benefit of pseudo-label guidance.
• The full UPFlow model with both components achieves the best performance across all evaluated benchmarks, including Sintel, MPI-Sintel, KITTI 2012, and KITTI 2015.