[Paper Review] Bilateral Grid Learning for Stereo Matching Networks

Bin Xu, Yuhua Xu+3 more|arXiv (Cornell University)|Jan 1, 2021

Advanced Vision and Imaging43 references4 citations

TL;DR

This paper proposes a parameter-free, edge-preserving cost volume upsampling module based on learned bilateral grids, enabling real-time stereo matching networks to achieve high accuracy by performing 3D convolutions at low resolution and upsampling efficiently. The method accelerates existing networks like GCNet, PSMNet, and GANet by 4–29× while maintaining competitive accuracy, and enables BGNet, a real-time network, to achieve 39 fps on KITTI with state-of-the-art performance.

ABSTRACT

Real-time performance of stereo matching networks is important for many applications, such as automatic driving, robot navigation and augmented reality (AR). Although significant progress has been made in stereo matching networks in recent years, it is still challenging to balance real-time performance and accuracy. In this paper, we present a novel edge-preserving cost volume upsampling module based on the slicing operation in the learned bilateral grid. The slicing layer is parameter-free, which allows us to obtain a high quality cost volume of high resolution from a low-resolution cost volume under the guide of the learned guidance map efficiently. The proposed cost volume upsampling module can be seamlessly embedded into many existing stereo matching networks, such as GCNet, PSMNet, and GANet. The resulting networks are accelerated several times while maintaining comparable accuracy. Furthermore, we design a real-time network (named BGNet) based on this module, which outperforms existing published real-time deep stereo matching networks, as well as some complex networks on the KITTI stereo datasets. The code is available at https://github.com/YuhuaXu/BGNet.

Motivation & Objective

• To address the trade-off between real-time performance and accuracy in stereo matching networks.
• To develop an efficient, parameter-free cost volume upsampling method that preserves edges and maintains high-resolution detail.
• To enable existing stereo matching networks to run significantly faster without sacrificing accuracy by offloading costly 3D convolutions to low-resolution cost volumes.
• To design a real-time stereo matching network (BGNet) that outperforms existing real-time and complex networks on KITTI benchmarks.
• To evaluate generalization performance on synthetic and real-world datasets, demonstrating robustness across domains.

Proposed method

• Proposes a novel cost volume upsampling module using slicing operations in a learned bilateral grid to upsample low-resolution cost volumes to high resolution.
• Leverages a differentiable bilateral grid to learn a guidance map that preserves edges during upsampling, avoiding blurring in textureless or occluded regions.
• The slicing layer is parameter-free, enabling fast inference with minimal computational overhead.
• Integrates the upsampling module into existing stereo matching networks (e.g., GCNet, PSMNet, GANet), allowing them to perform 3D convolutions at low resolution and upsample efficiently.
• Designs BGNet and BGNet+ as real-time stereo matching networks using the proposed module, optimized for inference speed and accuracy on KITTI datasets.
• Trains models on synthetic datasets (SceneFlow, IRS) to improve generalization without increasing computational cost.

Experimental results

Research questions

• RQ1 Can a parameter-free, edge-preserving upsampling method based on learned bilateral grids improve the efficiency of stereo matching networks without degrading accuracy?
• RQ2 Can the proposed upsampling module be seamlessly integrated into existing stereo matching architectures to accelerate inference while maintaining performance?
• RQ3 Can a real-time stereo matching network be designed using this module that outperforms both real-time and complex state-of-the-art networks on KITTI benchmarks?
• RQ4 How does the generalization performance of the proposed network compare to complex models when trained only on synthetic data?
• RQ5 To what extent does incorporating additional synthetic data (e.g., IRS) improve generalization without increasing inference time?

Key findings

• The proposed cost volume upsampling module accelerates existing stereo matching networks by 4 to 29 times while maintaining comparable accuracy to their original versions.
• BGNet achieves 39 fps on KITTI 2012 and KITTI 2015, outperforming all published real-time deep stereo matching networks and several complex networks like GCNet, AANet, and FADNet.
• On the Middlebury 2014 dataset, BGNet and BGNet+ achieve Bad 2.0 errors of 17.5% and 17.2% respectively when trained only on SceneFlow, outperforming iResNet and GANet.
• When trained on additional synthetic data (IRS), BGNet and BGNet+ reduce Bad 2.0 error to 13.5% and 11.6% respectively, demonstrating strong generalization with no increase in computational cost.
• The bilateral grid upsampling module is highly efficient, consuming only 4.3 ms per image pair on KITTI 2015, and is only 0.2 ms slower than linear upsampling in full-network GPU inference.
• The module is the first to apply differentiable bilateral grid operations in deep stereo matching, enabling high-quality, edge-preserving upsampling without additional parameters.