[Paper Review] MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction

Zehao Yu, Songyou Peng+3 more|arXiv (Cornell University)|Jun 1, 2022

Advanced Vision and Imaging166 citations

TL;DR

MonoSDF integrates monocular depth and normal cues into neural implicit surface reconstruction, improving accuracy and convergence across diverse scenes and representations (MLP and grid-based).

ABSTRACT

In recent years, neural implicit surface reconstruction methods have become popular for multi-view 3D reconstruction. In contrast to traditional multi-view stereo methods, these approaches tend to produce smoother and more complete reconstructions due to the inductive smoothness bias of neural networks. State-of-the-art neural implicit methods allow for high-quality reconstructions of simple scenes from many input views. Yet, their performance drops significantly for larger and more complex scenes and scenes captured from sparse viewpoints. This is caused primarily by the inherent ambiguity in the RGB reconstruction loss that does not provide enough constraints, in particular in less-observed and textureless areas. Motivated by recent advances in the area of monocular geometry prediction, we systematically explore the utility these cues provide for improving neural implicit surface reconstruction. We demonstrate that depth and normal cues, predicted by general-purpose monocular estimators, significantly improve reconstruction quality and optimization time. Further, we analyse and investigate multiple design choices for representing neural implicit surfaces, ranging from monolithic MLP models over single-grid to multi-resolution grid representations. We observe that geometric monocular priors improve performance both for small-scale single-object as well as large-scale multi-object scenes, independent of the choice of representation.

Motivation & Objective

• Motivate robust 3D reconstruction when multi-view cues are limited or textureless.
• Investigate how monocular depth and normal predictions can constrain neural implicit surfaces.
• Systematically compare neural implicit representations (MLP, dense grid, single/multi-resolution grids).
• Evaluate MonoSDF on object-level and room/scene-scale datasets to assess scalability and efficiency.

Proposed method

• Represent scene geometry as an SDF parameterized by four options: dense SDF grid, single MLP, single-resolution feature grid, and multi-resolution feature grids.
• Render via differentiable volume rendering to obtain RGB, depth, and normals for optimization.
• Predict monocular cues (depth and normals) with a pretrained Omnidata model and use them as supervision signals.
• Define loss combining RGB reconstruction, Eikonal regularization, depth consistency, and normal consistency with per-batch scale/shift alignment.
• Optimize jointly over scene representation and appearance using Adam with cue-enhanced losses.

Experimental results

Research questions

• RQ1 Can monocular depth and normal cues improve neural implicit surface reconstruction in textured-sparse or multi-object scenes?
• RQ2 Which neural implicit representations (MLP vs grid-based) benefit most from monocular priors in terms of quality and convergence speed?
• RQ3 How do depth and normal priors interact and complement RGB-based supervision across different scene scales?

Key findings

• Monocular cues significantly improve reconstruction quality for both MLP and multi-resolution grids, with best results when using both depth and normal cues.
• Multi-resolution grids outperform others in grid-based representations on Replica without cues, but with cues, both architectures improve and convergence speeds increase.
• MLP with monocular cues achieves the strongest overall results in many challenging settings, despite slower convergence than grids.
• On ScanNet, the MLP variant outperforms baselines, producing smoother reconstructions with more detail.
• DTU sparse-view experiments show that cues enable strong performance for both MLP and grids, with grids benefiting more from cues in dense views.
• multimodal evaluation indicates monocular priors enable scalable, robust reconstruction across object-, room-, and scene-level data.