[Paper Review] BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networks

Yao Yao, Zixin Luo+6 more|arXiv (Cornell University)|Nov 22, 2019

Advanced Vision and ImagingComputer Science35 references22 citations

TL;DR

This paper introduces BlendedMVS, a large-scale synthetic dataset for training generalized multi-view stereo (MVS) networks. By reconstructing high-quality textured 3D meshes from real images and rendering them into novel viewpoints, the authors generate training data where rendered color images are blended with input images to preserve realistic lighting while ensuring consistency with depth maps. Models trained on BlendedMVS show significantly improved generalization over existing datasets.

ABSTRACT

While deep learning has recently achieved great success on multi-view stereo (MVS), limited training data makes the trained model hard to be generalized to unseen scenarios. Compared with other computer vision tasks, it is rather difficult to collect a large-scale MVS dataset as it requires expensive active scanners and labor-intensive process to obtain ground truth 3D structures. In this paper, we introduce BlendedMVS, a novel large-scale dataset, to provide sufficient training ground truth for learning-based MVS. To create the dataset, we apply a 3D reconstruction pipeline to recover high-quality textured meshes from images of well-selected scenes. Then, we render these mesh models to color images and depth maps. To introduce the ambient lighting information during training, the rendered color images are further blended with the input images to generate the training input. Our dataset contains over 17k high-resolution images covering a variety of scenes, including cities, architectures, sculptures and small objects. Extensive experiments demonstrate that BlendedMVS endows the trained model with significantly better generalization ability compared with other MVS datasets. The dataset and pretrained models are available at \url{https://github.com/YoYo000/BlendedMVS}.

Motivation & Objective

• To address the lack of large-scale, diverse, and realistic training data for learning-based multi-view stereo (MVS) networks.
• To improve model generalization by incorporating realistic ambient lighting and visual cues from real images into synthetic training data.
• To provide a scalable, low-cost pipeline for generating high-quality training data without relying on expensive active scanners.
• To enable better performance on real-world and diverse scenes beyond small, fixed-trajectory datasets like DTU.
• To support future research in 3D geometry tasks by offering rich annotations including depth, occlusion, and normal maps.

Proposed method

• Reconstruct high-fidelity textured 3D meshes from real images using a 3D reconstruction pipeline.
• Render the textured meshes from multiple viewpoints to generate synthetic color images and corresponding depth maps.
• Blend the rendered color images with the original input images to preserve real-world lighting and texture while maintaining alignment with depth maps.
• Use the blended images as input for training MVS networks, ensuring consistency between appearance and geometry.
• Apply online photometric augmentation during training to further enhance robustness to view-dependent lighting effects.
• Ensure data privacy by removing or blurring dynamic elements (e.g., people) in the input images before blending.

Experimental results

Research questions

• RQ1 Can a synthetic dataset with blended real and rendered images improve the generalization of MVS networks across diverse real-world scenes?
• RQ2 How does incorporating ambient lighting from real input images into rendered training data affect model performance?
• RQ3 Does training on a large-scale, diverse dataset like BlendedMVS lead to better performance on benchmark datasets such as DTU and Tanks and Temples?
• RQ4 To what extent do photometric augmentations improve robustness to lighting variations in MVS networks?
• RQ5 Can the proposed data generation pipeline be adapted to other 3D geometry tasks beyond MVS?

Key findings

• Models trained on BlendedMVS achieve significantly better generalization performance on the DTU, Tanks and Temples, and ETH3D benchmarks compared to models trained on other datasets.
• The ablation study shows that blended images with online photometric augmentation yield the lowest validation error on the DTU dataset, outperforming training with input images or rendered images alone.
• Training with input images alone produces satisfactory results, suggesting that the reconstructed 3D models are accurate enough to serve as semi-ground truth for training.
• The use of blended images effectively preserves privacy by blurring or removing dynamic elements such as pedestrians from the training data.
• The dataset includes over 17,000 high-resolution images across 113 diverse scenes, including cities, architecture, sculptures, and small objects.
• The dataset supports additional annotations such as occlusion and normal maps, which can benefit future visibility-aware and patch-based MVS networks.