[Paper Review] Self-Supervised Learning for Stereo Matching with Self-Improving Ability

Yiran Zhong, Yuchao Dai+1 more|arXiv (Cornell University)|Sep 4, 2017

Advanced Vision and ImagingComputer Science9 references154 citations

TL;DR

The paper presents a self-supervised end-to-end stereo matching network that learns dense disparities from stereo pairs without ground-truth maps, using image warping losses and loop-consistency, with online self-improvement in new environments.

ABSTRACT

Exiting deep-learning based dense stereo matching methods often rely on ground-truth disparity maps as the training signals, which are however not always available in many situations. In this paper, we design a simple convolutional neural network architecture that is able to learn to compute dense disparity maps directly from the stereo inputs. Training is performed in an end-to-end fashion without the need of ground-truth disparity maps. The idea is to use image warping error (instead of disparity-map residuals) as the loss function to drive the learning process, aiming to find a depth-map that minimizes the warping error. While this is a simple concept well-known in stereo matching, to make it work in a deep-learning framework, many non-trivial challenges must be overcome, and in this work we provide effective solutions. Our network is self-adaptive to different unseen imageries as well as to different camera settings. Experiments on KITTI and Middlebury stereo benchmark datasets show that our method outperforms many state-of-the-art stereo matching methods with a margin, and at the same time significantly faster.

Motivation & Objective

• Address the reliance on ground-truth disparity maps in stereo matching.
• Propose an end-to-end network that learns dense disparities from stereo inputs without supervision.
• Introduce image warping-based losses and 3D feature regularization to avoid trivial solutions.
• Enable self-improvement by online adaptation to unseen imageries and camera settings.

Proposed method

• Extracts local features with a residual 18-layer 3x3 CNN to produce 64-channel features.
• Constructs cross-view feature volumes by concatenating left-right features across a disparity range.
• Performs 3D feature matching with a Residually connected Top-Down Module (Res-TDM) to produce a 3D disparity volume.
• Projects the 3D volume to a 2D disparity map using a soft-argmin operation.
• Evaluates warping quality between left/right images and uses photometric, gradient, and SSIM-based losses as supervision.
• Incorporates loop-consistency loss and a Maximum-Depth Heuristic to handle textureless regions and enforce symmetric constraints.

Experimental results

Research questions

• RQ1 Can dense stereo disparities be learned without ground-truth depth maps using self-supervision?
• RQ2 Does image reconstruction (warping) error suffice to supervise end-to-end stereo learning?
• RQ3 How can regularization in 3D feature space and loop-consistency improve performance in textureless regions?
• RQ4 Can the model adapt online to unseen scenes and camera settings without labeled data?

Key findings

• The self-supervised network achieves competitive performance on KITTI and Middlebury benchmarks without ground-truth disparities.
• The model demonstrates self-improving (online adaptation) ability when exposed to new stereo data.
• Inference time is practical for stereo pairs, with optional online fine-tuning increasing runtime.
• Loop-consistency and 3D regularization help mitigate degenerate solutions in textureless or challenging regions.
• The method adapts from KITTI-trained models to Middlebury with notable improvements after online iterations.