[Paper Review] Wasserstein Distances for Stereo Disparity Estimation

Divyansh Garg, Yan Wang+4 more|arXiv (Cornell University)|Jul 6, 2020

Advanced Vision and ImagingComputer Science58 references46 citations

TL;DR

Introduces a continuous disparity network (CDN) that outputs a distribution over disparity values using per-disparity offsets and trains with Wasserstein distance, improving disparity/depth estimation and downstream 3D object detection.

ABSTRACT

Existing approaches to depth or disparity estimation output a distribution over a set of pre-defined discrete values. This leads to inaccurate results when the true depth or disparity does not match any of these values. The fact that this distribution is usually learned indirectly through a regression loss causes further problems in ambiguous regions around object boundaries. We address these issues using a new neural network architecture that is capable of outputting arbitrary depth values, and a new loss function that is derived from the Wasserstein distance between the true and the predicted distributions. We validate our approach on a variety of tasks, including stereo disparity and depth estimation, and the downstream 3D object detection. Our approach drastically reduces the error in ambiguous regions, especially around object boundaries that greatly affect the localization of objects in 3D, achieving the state-of-the-art in 3D object detection for autonomous driving. Our code will be available at https://github.com/Div99/W-Stereo-Disp.

Motivation & Objective

• Motivate accurate depth/disparity estimation beyond discrete disparity bins and improve boundary Regions where depth is ambiguous.
• Propose a neural network that outputs continuous disparity values via offsets added to a discrete disparity set.
• Develop a Wasserstein-distance-based loss to align predicted distributions with ground-truth disparity distributions.
• Enable multi-modal ground-truth handling to capture ambiguity in depth at object boundaries.

Proposed method

• Introduce a continuous disparity network (CDN) that outputs a distribution over disparities by predicting an offset for each discrete disparity value.
• Replace the standard regression loss with a Wasserstein distance-based loss to directly match the predicted disparity distribution to the ground-truth distribution.
• Add an offset sub-network that, for each discrete disparity value, predicts a real-valued offset to shift mass and yield a continuous distribution.
• Represent ground-truth disparities as (potentially multi-modal) distributions and compute Wasserstein distance (W1 or W2) to train the model.
• Allow multi-modal ground truths by constructing distributions from neighborhoods and use 1D Wasserstein computation or CDF-based 1D formulations for training.

Experimental results

Research questions

• RQ1 Can a neural network output a continuous disparity distribution rather than a single integer disparity value?
• RQ2 Does learning with Wasserstein distance improve accuracy, especially at object boundaries where depth is ambiguous?
• RQ3 How does introducing offset predictions per disparity bin affect modality handling and convergence?
• RQ4 What is the impact of multi-modal ground truths on training efficiency and accuracy in depth/disparity estimation?

Key findings

• CDN achieves lower disparity errors than baselines on Scene Flow and KITTI 2015, particularly for foreground regions.
• The mode-based prediction with offsets and Wasserstein loss improves boundary pixel estimates, reducing multi-modal ambiguity.
• Multi-modal ground truth training speeds convergence and improves boundary accuracy.
• Disparity results show CDN variants outperform baseline PSMNet and GANet Deep on several metrics; downstream 3D object detection gains are substantial when using CDN-based depth.
• MM training facilitates faster convergence and better handling of inherently multi-modal disparities at boundaries.
• Disparity on object boundaries improves with CDN, yielding sharper foreground-background delineation in qualitative results.