[Paper Review] Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches

Jure Žbontar, Yann LeCun|arXiv (Cornell University)|Oct 20, 2015

Advanced Vision and ImagingComputer Science32 references939 citations

TL;DR

This paper proposes a convolutional neural network (CNN)-based method for stereo matching that learns a similarity measure between image patches to compute matching costs. By training on labeled patch pairs with known disparities, the CNN outperforms prior methods on KITTI 2012, KITTI 2015, and Middlebury datasets, achieving state-of-the-art accuracy with a fast architecture enabling real-time performance.

ABSTRACT

We present a method for extracting depth information from a rectified image pair. Our approach focuses on the first stage of many stereo algorithms: the matching cost computation. We approach the problem by learning a similarity measure on small image patches using a convolutional neural network. Training is carried out in a supervised manner by constructing a binary classification data set with examples of similar and dissimilar pairs of patches. We examine two network architectures for this task: one tuned for speed, the other for accuracy. The output of the convolutional neural network is used to initialize the stereo matching cost. A series of post-processing steps follow: cross-based cost aggregation, semiglobal matching, a left-right consistency check, subpixel enhancement, a median filter, and a bilateral filter. We evaluate our method on the KITTI 2012, KITTI 2015, and Middlebury stereo data sets and show that it outperforms other approaches on all three data sets.

Motivation & Objective

• To improve stereo matching by learning a robust similarity measure between image patches using deep learning.
• To address the challenge of matching cost computation—the first stage of stereo matching—by replacing hand-crafted metrics with a learned CNN-based approach.
• To develop two CNN architectures: one optimized for speed and another for accuracy, enabling trade-offs between performance and inference time.
• To demonstrate generalization across diverse stereo datasets, including KITTI 2012, KITTI 2015, and Middlebury, using transfer learning.
• To provide a fully reproducible pipeline with source code for training, inference, and post-processing steps.

Proposed method

• Train a CNN to classify pairs of image patches as similar or dissimilar using a binary cross-entropy loss on a supervised dataset with ground-truth disparities.
• Use two architectures: a fast network with fewer layers and filters for real-time inference, and an accurate network with deeper architecture and larger receptive fields.
• Initialize the stereo matching cost using the CNN’s output, which computes similarity scores between corresponding patches in left and right images.
• Apply cross-based cost aggregation to smooth costs using local image intensity consistency.
• Use semiglobal matching (SGM) with left-right consistency checks to enforce smoothness and eliminate erroneous disparities.
• Apply subpixel refinement, median filtering, and bilateral filtering to produce the final dense disparity map.

Experimental results

Research questions

• RQ1 Can a deep CNN learn a superior similarity measure for stereo matching compared to traditional hand-crafted metrics like SSD or NCC?
• RQ2 How does the performance of the CNN-based matching cost generalize across diverse stereo datasets such as KITTI and Middlebury?
• RQ3 What is the trade-off between accuracy and inference speed when varying CNN architecture hyperparameters?
• RQ4 How does the size of the training dataset affect the generalization and validation error of the CNN-based stereo method?
• RQ5 To what extent can a CNN trained on one dataset (e.g., Middlebury) be transferred to another (e.g., KITTI) without fine-tuning?

Key findings

• The CNN-based method achieved the lowest error rate on the KITTI 2012, KITTI 2015, and Middlebury stereo benchmarks, outperforming all previously published methods.
• The accurate CNN architecture reduced the average end-point error (AEE) on KITTI 2012 to 2.97 pixels and on KITTI 2015 to 3.23 pixels, with the best configuration achieving 2.60 and 3.23 respectively.
• The fast architecture achieved up to 90× faster inference than the accurate version while increasing error by only ~0.5 pixels, demonstrating strong real-time potential.
• Transfer learning showed that a CNN trained on Middlebury generalizes well to KITTI, achieving validation error comparable to models trained on KITTI itself.
• Increasing network depth and width improved generalization up to a point, after which performance degraded due to data limitations, indicating a saturation effect.
• Hyperparameter search revealed that optimal performance was achieved with 128 feature maps, 5 convolutional layers, 4 fully connected layers, and specific negative sampling ratios (dataset_neg_low=1.5, dataset_neg_high=18).