[Paper Review] PDC-Net+: Enhanced Probabilistic Dense Correspondence Network

Prune Truong, Martin Danelljan+2 more|arXiv (Cornell University)|Sep 28, 2021

Human Pose and Action Recognition4 citations

TL;DR

PDC-Net+ proposes a probabilistic deep network for dense correspondence estimation that jointly predicts optical flow and uncertainty using a constrained mixture of Laplace distributions, enabling robust confidence estimation under large displacements, occlusions, and appearance changes. It achieves state-of-the-art performance on geometric matching and optical flow benchmarks while significantly improving uncertainty calibration for downstream tasks like 3D reconstruction and pose estimation.

ABSTRACT

Establishing robust and accurate correspondences between a pair of images is a long-standing computer vision problem with numerous applications. While classically dominated by sparse methods, emerging dense approaches offer a compelling alternative paradigm that avoids the keypoint detection step. However, dense flow estimation is often inaccurate in the case of large displacements, occlusions, or homogeneous regions. In order to apply dense methods to real-world applications, such as pose estimation, image manipulation, or 3D reconstruction, it is therefore crucial to estimate the confidence of the predicted matches. We propose the Enhanced Probabilistic Dense Correspondence Network, PDC-Net+, capable of estimating accurate dense correspondences along with a reliable confidence map. We develop a flexible probabilistic approach that jointly learns the flow prediction and its uncertainty. In particular, we parametrize the predictive distribution as a constrained mixture model, ensuring better modelling of both accurate flow predictions and outliers. Moreover, we develop an architecture and an enhanced training strategy tailored for robust and generalizable uncertainty prediction in the context of self-supervised training. Our approach obtains state-of-the-art results on multiple challenging geometric matching and optical flow datasets. We further validate the usefulness of our probabilistic confidence estimation for the tasks of pose estimation, 3D reconstruction, image-based localization, and image retrieval. Code and models are available at https://github.com/PruneTruong/DenseMatching.

Motivation & Objective

• To address the lack of reliable uncertainty estimation in dense correspondence networks, especially under large displacements, occlusions, and appearance variations.
• To develop a self-supervised training framework that generalizes well to real-world geometric matching scenarios without requiring dense annotations.
• To enable dense correspondence methods to support applications traditionally reliant on sparse keypoint matching, such as pose estimation and image-based localization.
• To improve the robustness of uncertainty prediction by modeling both inlier and outlier flow predictions explicitly through a probabilistic formulation.
• To design an architecture and training strategy that prevent overconfidence in texture-less or occluded regions during self-supervised learning.

Proposed method

• The network models the conditional flow distribution as a constrained mixture of two Laplace components, where the first component captures inlier predictions and the second models outliers.
• The mixture components are constrained via fixed lower and upper bounds on variances (σ²₁ = 1.0, 2.0 ≤ σ²₂ ≤ HW), resolving permutation ambiguity and improving uncertainty separation.
• An independent spatial processing module is introduced in the correlation volume decoder to prevent over-extrapolation and reduce overconfidence in texture-less regions.
• An injective training mask is applied during self-supervised learning to enforce injectivity of ground-truth flow, enabling better generalization to occluded and moving object regions.
• A multi-object training data generation strategy is used to simulate independently moving objects, improving robustness to complex scene dynamics.
• The model is trained end-to-end using a differentiable loss that optimizes both flow accuracy (AEPE, Fl) and uncertainty calibration (AUSE), with uncertainty estimated as the probability of being an outlier.

Experimental results

Research questions

• RQ1 Can a deep network jointly predict dense optical flow and reliable uncertainty estimates in challenging geometric matching scenarios?
• RQ2 How can uncertainty be modeled effectively to distinguish between inlier predictions and outliers in the presence of large displacements and occlusions?
• RQ3 What architectural and training modifications are necessary to prevent overconfidence in self-supervised dense correspondence networks?
• RQ4 To what extent can self-supervised training with realistic scene dynamics improve generalization and uncertainty calibration?
• RQ5 Can probabilistic confidence maps from dense networks enable state-of-the-art performance in downstream tasks like pose estimation and 3D reconstruction?

Key findings

• PDC-Net+ achieves state-of-the-art performance on the KITTI-2015, MegaDepth, and YFCC100M datasets, with 5.55 AEPE and 16.75% Fl on KITTI-2015.
• The model reduces AUSE (uncertainty calibration error) to 0.136 on KITTI-2015 and 0.238 on YFCC100M, indicating superior uncertainty calibration compared to baselines.
• Training with multiple independently moving objects improves flow accuracy by 3% in Fl on KITTI-2015, demonstrating enhanced robustness to complex motion.
• The injective training mask leads to a 5.55 AEPE on KITTI-2015, outperforming both no mask and occlusion mask strategies, especially in occluded regions.
• On YFCC100M, PDC-Net+ achieves 44.45% mAP@5° and 54.70% mAP@10° in image-based localization, showing strong transferability to real-world applications.
• The constrained mixture model with fixed variance bounds (σ²₁ = 1.0, 2.0 ≤ σ²₂ ≤ HW) yields the most stable training and best overall performance, outperforming unconstrained or fixed-variance alternatives.