[Paper Review] Neural 3D Scene Reconstruction with the Manhattan-world Assumption

Haoyu Guo, Sida Peng+5 more|arXiv (Cornell University)|May 5, 2022

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes a novel neural 3D scene reconstruction method that integrates the Manhattan-world assumption into implicit neural representations to improve geometry reconstruction in low-textured indoor scenes. By jointly optimizing signed distance functions and semantic labels in 3D space using a novel loss, the method achieves superior reconstruction quality and semantic consistency, especially on planar regions, outperforming state-of-the-art methods on ScanNet and 7-Scenes.

ABSTRACT

This paper addresses the challenge of reconstructing 3D indoor scenes from multi-view images. Many previous works have shown impressive reconstruction results on textured objects, but they still have difficulty in handling low-textured planar regions, which are common in indoor scenes. An approach to solving this issue is to incorporate planer constraints into the depth map estimation in multi-view stereo-based methods, but the per-view plane estimation and depth optimization lack both efficiency and multi-view consistency. In this work, we show that the planar constraints can be conveniently integrated into the recent implicit neural representation-based reconstruction methods. Specifically, we use an MLP network to represent the signed distance function as the scene geometry. Based on the Manhattan-world assumption, planar constraints are employed to regularize the geometry in floor and wall regions predicted by a 2D semantic segmentation network. To resolve the inaccurate segmentation, we encode the semantics of 3D points with another MLP and design a novel loss that jointly optimizes the scene geometry and semantics in 3D space. Experiments on ScanNet and 7-Scenes datasets show that the proposed method outperforms previous methods by a large margin on 3D reconstruction quality. The code is available at https://zju3dv.github.io/manhattan_sdf.

Motivation & Objective

• Address the challenge of poor 3D reconstruction in low-textured planar regions (e.g., floors and walls) common in indoor scenes.
• Overcome the limitations of multi-view stereo (MVS) methods that optimize depth maps per view, leading to inconsistent depth and plane estimates.
• Leverage the Manhattan-world assumption to enforce geometric consistency in planar regions during implicit neural representation learning.
• Improve semantic segmentation accuracy by jointly optimizing 3D semantics with geometry, reducing noise and inconsistency across views.
• Achieve globally consistent 3D reconstruction and segmentation by learning from multi-view images in a unified implicit representation framework.

Proposed method

• Represent 3D scenes using a multi-layer perceptron (MLP) that predicts signed distance function (SDF), color, and semantic logits for any 3D point.
• Initialize semantic logits using a 2D semantic segmentation network (e.g., DeepLabV3+) applied to input images.
• Enforce the Manhattan-world assumption by regularizing surface normals of floor and wall regions to align with three dominant orthogonal directions.
• Introduce a novel joint optimization loss that simultaneously optimizes SDF geometry and semantic logits in 3D space to correct inaccurate 2D segmentation.
• Use differentiable volume rendering to compare rendered images with input images, enabling end-to-end training based on photometric consistency.
• Apply multi-view consistency during training by optimizing geometry and semantics jointly across all input views.

Experimental results

Research questions

• RQ1 Can the Manhattan-world assumption be effectively integrated into implicit neural representation-based 3D reconstruction to improve planar region reconstruction?
• RQ2 How does joint optimization of 3D geometry and semantics enhance reconstruction quality and segmentation accuracy compared to separate optimization?
• RQ3 To what extent does the proposed method reduce noise and inconsistency in semantic segmentation across multiple views?
• RQ4 Does the joint optimization of geometry and semantics lead to more accurate and complete reconstruction of low-textured planar regions?
• RQ5 How does the method compare to state-of-the-art MVS and neural rendering methods in terms of reconstruction precision, recall, and novel view synthesis quality?

Key findings

• The proposed method achieves state-of-the-art 3D reconstruction quality on the ScanNet and 7-Scenes datasets, with significant improvements in both precision and recall, especially in planar regions.
• On ScanNet, the method outperforms COLMAP, NeRF, UNISURF, NeuS, and VolSDF in terms of both reconstruction accuracy and completeness, particularly in texture-less areas.
• Semantic segmentation IoU for floor regions improved from 0.532 (DeepLabV3+) to 0.624, and for wall regions from 0.475 to 0.518, demonstrating substantial gains in segmentation consistency and accuracy.
• The joint optimization of semantics and geometry reduces noisy and inconsistent segmentation labels across views, especially near object boundaries.
• Novel view synthesis results show that the method produces higher-quality images than NeRF and VolSDF, even in views far from training views, due to improved geometric fidelity.
• The method maintains high-fidelity details in non-planar regions while significantly enhancing planar structure reconstruction, demonstrating robustness and generalization.