[Paper Review] Pano Popups: Indoor 3D Reconstruction with a Plane-Aware Network

Marc Eder, Pierre Moulon+1 more|arXiv (Cornell University)|Jul 1, 2019

Advanced Vision and Imaging29 references4 citations

TL;DR

This paper proposes a plane-aware convolutional neural network that predicts depth, surface normals, and plane boundaries from a single $360^\circ$ indoor image, using a novel loss function based on principal curvature to improve geometric consistency. The method achieves state-of-the-art performance in monocular omnidirectional depth and normal estimation and enables high-quality 3D 'pop-up' reconstructions by combining predictions with geodesic coordinate maps as priors.

ABSTRACT

In this work we present a method to train a plane-aware convolutional neural network for dense depth and surface normal estimation as well as plane boundaries from a single indoor $360^\circ$ image. Using our proposed loss function, our network outperforms existing methods for single-view, indoor, omnidirectional depth estimation and provides an initial benchmark for surface normal prediction from $360^\circ$ images. Our improvements are due to the use of a novel plane-aware loss that leverages principal curvature as an indicator of planar boundaries. We also show that including geodesic coordinate maps as network priors provides a significant boost in surface normal prediction accuracy. Finally, we demonstrate how we can combine our network's outputs to generate high quality 3D "pop-up" models of indoor scenes.

Motivation & Objective

• To address the challenge of single-view 3D reconstruction in indoor scenes using $360^\circ$ omnidirectional images, which offer a wider field of view than perspective images.
• To improve depth and surface normal estimation accuracy for omnidirectional images by incorporating geometric priors such as planarity and curvature.
• To develop a method that generates high-quality 3D planar models ('pop-up' models) from a single $360^\circ$ image using predicted depth, normals, and plane boundaries.
• To demonstrate that geodesic coordinate maps serve as effective priors for enhancing surface normal prediction in spherical image representations.
• To establish a benchmark for plane boundary detection and surface normal prediction from $360^\circ$ images, which are underexplored in existing literature.

Proposed method

• A multi-scale encoder-decoder CNN architecture is used, with separate decoder branches for depth and for surface normals and curvature prediction.
• A novel plane-aware loss function is introduced that modulates each pixel’s contribution to the loss based on its principal curvature, promoting planar consistency in predictions.
• Geodesic coordinate maps are used as additional input channels to help the network learn spatial relationships on the spherical image surface, improving normal estimation.
• Plane boundaries are predicted via a separate head that detects discontinuities in surface normals, with Otsu thresholding used to segment connected components for 3D model generation.
• 3D planar models are reconstructed by estimating plane equations using RANSAC with normal and depth information, followed by back-projection and meshing on an icosahedral grid.
• The network is trained end-to-end using a composite loss combining depth, normal, and curvature supervision, with curvature derived from predicted normals.

Figure 1 : Visualization of our paper’s contributions. Given an RGB omnidirectional image (top), we predict depth, surface normals, and plane boundary maps (middle) with state-of-the-art accuracy. Then we show we can use this information to achieve a planar segmentation and 3D reconstruction of the

Experimental results

Research questions

• RQ1 Can a plane-aware loss function based on principal curvature improve depth and surface normal estimation in $360^\circ$ images?
• RQ2 Does incorporating geodesic coordinate maps as input priors enhance the accuracy of surface normal prediction for omnidirectional images?
• RQ3 To what extent can a single $360^\circ$ image support high-quality 3D planar reconstruction without stereo or multi-view data?
• RQ4 How does the proposed method compare to existing monocular depth estimation networks when applied to omnidirectional images?
• RQ5 Can the network’s outputs be effectively combined to generate sharp-edged, planar 3D 'pop-up' models from a single input image?

Key findings

• The proposed plane-aware loss function significantly improves depth and surface normal estimation accuracy, achieving state-of-the-art performance on $360^\circ$ indoor depth estimation benchmarks.
• Including geodesic coordinate maps as input channels leads to a measurable improvement in surface normal prediction, reducing errors on large planar surfaces such as walls.
• The network produces high-quality plane boundary maps that enable robust, threshold-free segmentation of planar regions in the scene.
• 3D 'pop-up' models generated from the network’s outputs exhibit sharp, well-defined plane boundaries, outperforming baseline depth-only methods that produce smoothed, noisy reconstructions.
• The integration of surface normals into the RANSAC plane fitting process results in significantly more accurate and stable 3D reconstructions compared to depth-only fitting.
• The method demonstrates that a single $360^\circ$ image can support detailed, piecewise-planar 3D reconstruction when combined with geometric priors and curvature-aware learning.

Figure 2 : Overview of our network architecture. The vectors next to each layer are $\left[\begin{smallmatrix}\text{input features}\\ \text{output features}\end{smallmatrix}\right]$ for each. Above each layer is the kernel size, ‘T’ indicates transposed convolution, ‘s#’ indicates stride, and ‘d#’ i