[Paper Review] Learning Unsupervised Multi-View Stereopsis via Robust Photometric Consistency

Tejas Khot, Shubham Agrawal+4 more|arXiv (Cornell University)|May 7, 2019

Advanced Vision and Imaging36 references64 citations

TL;DR

The paper proposes an unsupervised learning approach for multi-view stereo (MVS) that uses a robust photometric consistency loss to train depth prediction without ground-truth 3D data, enabling competitive dense reconstructions with implicit occlusion handling.

ABSTRACT

We present a learning based approach for multi-view stereopsis (MVS). While current deep MVS methods achieve impressive results, they crucially rely on ground-truth 3D training data, and acquisition of such precise 3D geometry for supervision is a major hurdle. Our framework instead leverages photometric consistency between multiple views as supervisory signal for learning depth prediction in a wide baseline MVS setup. However, naively applying photo consistency constraints is undesirable due to occlusion and lighting changes across views. To overcome this, we propose a robust loss formulation that: a) enforces first order consistency and b) for each point, selectively enforces consistency with some views, thus implicitly handling occlusions. We demonstrate our ability to learn MVS without 3D supervision using a real dataset, and show that each component of our proposed robust loss results in a significant improvement. We qualitatively observe that our reconstructions are often more complete than the acquired ground truth, further showing the merits of this approach. Lastly, our learned model generalizes to novel settings, and our approach allows adaptation of existing CNNs to datasets without ground-truth 3D by unsupervised finetuning. Project webpage: https://tejaskhot.github.io/unsup_mvs

Motivation & Objective

• Motivate learning MVS without requiring ground-truth 3D supervision.
• Leverage photometric consistency across views as supervision for depth prediction.
• Develop a robust loss that handles occlusion and lighting changes in MVS settings.

Proposed method

• Predict per-image depth maps with a CNN given multiple views.
• Use differentiable warping to synthesize views and compute photometric reprojection losses.
• Introduce a robust per-pixel top-K aggregation of view losses to handle occlusion.
• Incorporate first-order gradient consistency to mitigate lighting changes.
• Combine photometric loss with SSIM and edge-aware depth smoothness in a weighted objective.
• Train end-to-end with ADAM on multi-view image sets; inference fuses depth maps into a 3D point cloud.

Experimental results

Research questions

• RQ1 Can MVS depth prediction be learned without 3D ground truth by leveraging photometric consistency across views?
• RQ2 How can occlusions and lighting changes be robustly handled in a multi-view photometric loss framework?
• RQ3 Does a top-K selection of view-consistency losses improve training signals for depth prediction in MVS?
• RQ4 Does the learned model generalize to unseen datasets and settings without 3D supervision?
• RQ5 Can unsupervised fine-tuning improve performance on datasets lacking 3D annotations?

Key findings

• Robust photometric loss with top-K view aggregation improves reconstruction quality over naive photo-consistency.
• The robust model outperforms baselines and several classical methods on DTU in several metrics, approaching supervised methods.
• Training with the robust loss yields fewer holes and more complete reconstructions in low-texture regions.
• Fine-tuning pretrained models on new datasets with the robust loss improves performance without 3D annotations.
• The model demonstrates generalization to Tanks and Temples without fine-tuning, albeit with limitations due to depth-range challenges.