[Paper Review] Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching

Xiaodong Gu, Zhiwen Fan+4 more|arXiv (Cornell University)|Dec 13, 2019

Advanced Vision and Imaging54 references65 citations

TL;DR

Introduces a cascade cost volume to enable high-resolution depth/disparity estimation with reduced memory and compute by progressively refining depth ranges and using multi-scale feature pyramids. Demonstrates state-of-the-art results on DTU and Tanks and Temples, with notable memory and speed gains.

ABSTRACT

The deep multi-view stereo (MVS) and stereo matching approaches generally construct 3D cost volumes to regularize and regress the output depth or disparity. These methods are limited when high-resolution outputs are needed since the memory and time costs grow cubically as the volume resolution increases. In this paper, we propose a both memory and time efficient cost volume formulation that is complementary to existing multi-view stereo and stereo matching approaches based on 3D cost volumes. First, the proposed cost volume is built upon a standard feature pyramid encoding geometry and context at gradually finer scales. Then, we can narrow the depth (or disparity) range of each stage by the depth (or disparity) map from the previous stage. With gradually higher cost volume resolution and adaptive adjustment of depth (or disparity) intervals, the output is recovered in a coarser to fine manner. We apply the cascade cost volume to the representative MVS-Net, and obtain a 23.1% improvement on DTU benchmark (1st place), with 50.6% and 74.2% reduction in GPU memory and run-time. It is also the state-of-the-art learning-based method on Tanks and Temples benchmark. The statistics of accuracy, run-time and GPU memory on other representative stereo CNNs also validate the effectiveness of our proposed method.

Motivation & Objective

• Address the memory and time bottlenecks of 3D cost volumes in high-resolution MVS and stereo matching.
• Propose a coarse-to-fine cascade cost volume built on a feature pyramid to progressively refine depth/disparity.
• Adaptive depth sampling driven by previous stage predictions to reduce computation.

Proposed method

• Construct a cascade of cost volumes across multiple stages with decreasing depth range Rk and decreasing plane interval Ik, such that Dk = Rk/Ik.
• Use a feature pyramid to provide multi-scale features and progressively increase spatial resolution across stages.
• Warp operations and residual depth/disparity deltas (Δk+1) guide the next-stage cost volumes using updated homographies or disparity mappings.
• Supervise all intermediate outputs with a multi-stage loss: Loss = sum_k λ^k L^k.
• Integrate cascade cost volumes with backbone networks like MVSNet and PSMNet to form MVSNet+Ours and stereo counterparts, enabling high-resolution outputs.

Experimental results

Research questions

• RQ1 Can a cascade, coarse-to-fine cost volume reduce memory and compute while preserving or improving depth/disparity accuracy at high resolutions?
• RQ2 How does progressive refinement using prior-stage depth estimates affect performance on MVS and stereo benchmarks?
• RQ3 What are the trade-offs between spatial resolution, number of stages, and hypothesis planes in cascade cost volumes?
• RQ4 How well does the cascade approach generalize across datasets (DTU, Tanks and Temples, KITTI, Scene Flow)?

Key findings

• MVSNet+Ours achieves first place on DTU when combined with MVSNet, with a 35.6% improvement in overall quality.
• The cascade cost volume reduces GPU memory by about 50.6% and run-time by about 59.3% on DTU when using the same total depth hypotheses.
• On Tanks and Temples, MVSNet+Ours achieves state-of-the-art performance among learning-based MVS methods without fine-tuning.
• In stereo matching, cascade cost volume reduces end-point-error (EPE) by up to 0.166 over baseline methods (e.g., PSMNet, GwcNet, GANet11) and lowers memory usage (e.g., PSMNet, GwcNet, GANet11 show notable memory savings).
• Table-level results show that with cascade, MVSNet's Acc., Comp., and Overall metrics improve (e.g., from 0.456/0.646/0.551 to 0.325/0.385/0.355 on DTU with 4–stage cascade in some configurations) and memory/runtime are substantially reduced.