[Paper Review] DiffuStereo: High Quality Human Reconstruction via Diffusion-based Stereo Using Sparse Cameras

Ruizhi Shao, Zerong Zheng+3 more|arXiv (Cornell University)|Jul 16, 2022

Advanced Vision and Imaging4 citations

TL;DR

DiffuStereo proposes a novel 3D human reconstruction system using only eight sparse RGB cameras by introducing a diffusion-based stereo module that iteratively refines depth maps with sub-pixel accuracy. The method achieves high-quality, detailed 3D reconstructions rivaling dense-camera rigs, outperforming state-of-the-art methods by over 42% in sub-1mm reconstruction accuracy and reducing Chamfer distance by 54.4% compared to coarse initial models.

ABSTRACT

We propose DiffuStereo, a novel system using only sparse cameras (8 in this work) for high-quality 3D human reconstruction. At its core is a novel diffusion-based stereo module, which introduces diffusion models, a type of powerful generative models, into the iterative stereo matching network. To this end, we design a new diffusion kernel and additional stereo constraints to facilitate stereo matching and depth estimation in the network. We further present a multi-level stereo network architecture to handle high-resolution (up to 4k) inputs without requiring unaffordable memory footprint. Given a set of sparse-view color images of a human, the proposed multi-level diffusion-based stereo network can produce highly accurate depth maps, which are then converted into a high-quality 3D human model through an efficient multi-view fusion strategy. Overall, our method enables automatic reconstruction of human models with quality on par to high-end dense-view camera rigs, and this is achieved using a much more light-weight hardware setup. Experiments show that our method outperforms state-of-the-art methods by a large margin both qualitatively and quantitatively.

Motivation & Objective

• To address the challenge of high-accuracy 3D human reconstruction from sparse-view RGB cameras, which current learning-based methods fail to achieve due to reliance on appearance cues rather than cross-view correspondence.
• To overcome limitations of existing stereo matching methods—such as discrete cost volumes and low resolution—by integrating diffusion models into iterative stereo matching for continuous, sub-pixel refinement.
• To design a memory-efficient, multi-level network architecture capable of processing high-resolution (up to 4K) inputs without excessive memory usage.
• To enable high-fidelity 3D reconstruction using lightweight hardware by combining diffusion-based stereo with a lightweight multi-view fusion strategy that handles calibration errors and occlusions.

Proposed method

• A novel diffusion-based stereo module is introduced, which uses a custom diffusion kernel and stereo constraints to refine disparity flows iteratively in a 2D flow domain.
• The method employs a two-level network architecture: a global level for semantic feature extraction from downsampled images and a diffusion level that integrates these features into a diffusion-based stereo network for iterative refinement.
• The diffusion network is conditioned on multi-view features and epipolar constraints to ensure stereo consistency and improve geometric accuracy under sparse-view settings.
• A patch-based training strategy is adopted in the two-level architecture to mitigate memory bottlenecks when processing high-resolution inputs.
• An efficient multi-view fusion strategy aligns refined depth point clouds with a coarse human model via non-rigid ICP and fuses depth maps while handling calibration errors and occlusions.
• The system starts from a coarse 3D human mesh generated by DoubleField, which is then used to render initial depth maps and disparity flows for diffusion refinement.

Experimental results

Research questions

• RQ1 Can diffusion models be effectively adapted to stereo matching for high-accuracy 3D human reconstruction from sparse-view RGB inputs?
• RQ2 How can diffusion-based stereo achieve sub-pixel accuracy and handle high-resolution (4K) inputs without prohibitive memory costs?
• RQ3 To what extent can a diffusion-based stereo module improve geometric detail and reconstruction accuracy compared to existing iterative stereo or implicit learning methods?
• RQ4 How robust is the method to initial model errors and calibration inaccuracies in real-world sparse-view setups?

Key findings

• Our method reduces the Chamfer distance by 54.4% and P2S error by 44.1% compared to the coarse model produced by DoubleField, demonstrating significant improvement in geometric accuracy.
• For sub-1mm reconstruction accuracy, our method outperforms previous state-of-the-art methods by more than 42%, proving its superiority in preserving fine geometric details.
• The proposed diffusion-based stereo network achieves high-quality depth maps with sub-pixel accuracy, even under sparse-view conditions with only eight cameras.
• Using only 30 reverse steps, the diffusion model produces high-quality results in about 12 seconds for eight views, achieving a 95% speedup over the original 1000-step diffusion model.
• The ablation study confirms that the proposed diffusion kernel is more stable and accurate than the original kernel, which produced noisy results in some cases.
• Even when initialized with a visual hull from eight views, the method successfully recovers fine geometric details in visible regions, showing robustness to coarse initial geometry.