[Paper Review] Self-supervised Learning of Depth Inference for Multi-view Stereo

Jiayu Yang, José M. Alvarez+1 more|arXiv (Cornell University)|Apr 7, 2021

Advanced Vision and Imaging27 references4 citations

TL;DR

This paper proposes a self-supervised learning framework for multi-view stereo depth estimation that generates high-quality pseudo depth labels through iterative refinement, achieving performance on par with supervised methods on the DTU dataset. The method uses unsupervised depth estimation from image reconstruction, refines pseudo labels via high-resolution inference and multi-view fusion, and iteratively improves network performance without ground-truth depth supervision.

ABSTRACT

Recent supervised multi-view depth estimation networks have achieved promising results. Similar to all supervised approaches, these networks require ground-truth data during training. However, collecting a large amount of multi-view depth data is very challenging. Here, we propose a self-supervised learning framework for multi-view stereo that exploit pseudo labels from the input data. We start by learning to estimate depth maps as initial pseudo labels under an unsupervised learning framework relying on image reconstruction loss as supervision. We then refine the initial pseudo labels using a carefully designed pipeline leveraging depth information inferred from higher resolution images and neighboring views. We use these high-quality pseudo labels as the supervision signal to train the network and improve, iteratively, its performance by self-training. Extensive experiments on the DTU dataset show that our proposed self-supervised learning framework outperforms existing unsupervised multi-view stereo networks by a large margin and performs on par compared to the supervised counterpart. Code is available at https://github.com/JiayuYANG/Self-supervised-CVP-MVSNet.

Motivation & Objective

• To address the challenge of acquiring large-scale multi-view depth data for training MVS networks.
• To develop a self-supervised learning framework that eliminates dependency on ground-truth depth labels.
• To improve the quality of pseudo depth labels generated from unsupervised image reconstruction for better supervision.
• To achieve performance comparable to supervised MVS networks using only multi-view images.
• To iteratively refine pseudo labels through multi-stage processing to boost network generalization.

Proposed method

• Initial depth maps are generated via unsupervised training using image reconstruction loss to enforce photometric consistency across views.
• High-resolution images are used to infer more accurate depth estimates for trustable pixels, improving initial pseudo label quality.
• Pseudo labels are filtered by leveraging depth consistency across neighboring views to reduce errors.
• Multi-view depth fusion and mesh generation are applied to complete incomplete pseudo labels and improve spatial coherence.
• The refined pseudo labels are used to supervise the network in an iterative self-training loop, improving performance over time.
• The framework integrates components from CVP-MVSNet and uses incremental self-supervision to progressively enhance depth estimation.

Experimental results

Research questions

• RQ1 Can a self-supervised learning framework generate high-quality pseudo depth labels without any ground-truth depth supervision?
• RQ2 How does refining initial unsupervised pseudo labels using high-resolution inference and multi-view fusion improve depth estimation accuracy?
• RQ3 To what extent can iterative self-training with refined pseudo labels close the performance gap between unsupervised and supervised MVS networks?
• RQ4 What are the limitations of the method in texture-less regions, and how can they be addressed?
• RQ5 Does the proposed refinement pipeline lead to stable and consistent performance improvements across training iterations?

Key findings

• The proposed self-supervised framework outperforms existing unsupervised MVS networks by a large margin on the DTU dataset.
• With iterative self-training, the method achieves an f-score of 88.42% after three iterations, only 0.12% lower than the supervised counterpart (88.61%).
• The model's performance stabilizes after the third iteration, indicating no degradation from continued self-supervision.
• The use of high-resolution pseudo label refinement and multi-view fusion significantly improves reconstruction quality, reducing overall error from 0.567 to 0.363.
• The method achieves state-of-the-art performance among unsupervised MVS networks, demonstrating strong generalization from only multi-view images.
• A key limitation remains in texture-less regions, where the initial unsupervised stage fails to generate any depth labels due to lack of photometric consistency.