[Paper Review] End-to-End Learning of Geometry and Context for Deep Stereo Regression

Alex Kendall, Hayk Martirosyan+5 more|arXiv (Cornell University)|Mar 13, 2017

Advanced Vision and Imaging35 references279 citations

TL;DR

GC-Net presents an end-to-end deep stereo regression framework that forms a differentiable cost volume using 3-D convolutions and a soft argmin, achieving sub-pixel disparities and state-of-the-art KITTI results without post-processing.

ABSTRACT

We propose a novel deep learning architecture for regressing disparity from a rectified pair of stereo images. We leverage knowledge of the problem's geometry to form a cost volume using deep feature representations. We learn to incorporate contextual information using 3-D convolutions over this volume. Disparity values are regressed from the cost volume using a proposed differentiable soft argmin operation, which allows us to train our method end-to-end to sub-pixel accuracy without any additional post-processing or regularization. We evaluate our method on the Scene Flow and KITTI datasets and on KITTI we set a new state-of-the-art benchmark, while being significantly faster than competing approaches.

Motivation & Objective

• Learn an end-to-end mapping from rectified stereo pairs to per-pixel disparities.
• Incorporate stereo geometry via a differentiable cost volume.
• Leverage semantic context through 3-D convolutions over height, width, and disparity.
• Regress disparities with sub-pixel accuracy using a differentiable soft argmin.
• Demonstrate performance on Scene Flow and benchmark on KITTI datasets.

Proposed method

• Learn unary feature representations from left and right images using a shared 2-D CNN with residual blocks.
• Construct a cost volume by concatenating left-right unary features across disparity levels while preserving feature dimensions.
• Apply a 3-D encoder-decoder to regularize the cost volume by aggregating height, width, and disparity context.
• Use a differentiable soft argmin to regress disparity values from the regularized cost volume.
• Train end-to-end with an L1 disparity loss (with optional regression vs. classification losses) to achieve sub-pixel accuracy.

Experimental results

Research questions

• RQ1 Can end-to-end learning effectively regress per-pixel disparities from rectified stereo pairs while leveraging geometric cost volumes?
• RQ2 Does incorporating wide contextual information via 3-D convolutions improve stereo regression accuracy and robustness?
• RQ3 Can a differentiable soft argmin provide accurate sub-pixel disparity estimates without post-processing?
• RQ4 Does end-to-end training enable the model to learn semantic and contextual cues for challenging scenes?

Key findings

• The model achieves state-of-the-art results on KITTI benchmarks, outperforming prior methods.
• 3-D contextual regularization of the cost volume significantly improves disparity accuracy over unary-based approaches.
• A regression loss yields better accuracy and sub-pixel results compared to hard or soft classification losses.
• End-to-end GC-Net outperforms patch-based deep stereo methods that require post-processing like SGM.
• The soft argmin enables differentiable sub-pixel disparity regression without additional post-processing.
• Qualitative results show the network utilizing wider contextual information to handle reflective, textureless, and thin structures.