[Paper Review] LED2-Net: Monocular 360 Layout Estimation via Differentiable Depth Rendering

Fu-En Wang, Yu-Hsuan Yeh+3 more|arXiv (Cornell University)|Apr 1, 2021

Advanced Vision and Imaging31 references4 citations

TL;DR

LED2-Net proposes a differentiable layout-to-depth (L2D) rendering method that formulates monocular 360° layout estimation as a 3D-aware depth estimation task on the horizon line, enabling end-to-end training with 3D geometric supervision. The method achieves state-of-the-art performance on multiple benchmarks and enables improved generalization via pre-training on synthetic depth datasets like Structured3D.

ABSTRACT

Although significant progress has been made in room layout estimation, most methods aim to reduce the loss in the 2D pixel coordinate rather than exploiting the room structure in the 3D space. Towards reconstructing the room layout in 3D, we formulate the task of 360 layout estimation as a problem of predicting depth on the horizon line of a panorama. Specifically, we propose the Differentiable Depth Rendering procedure to make the conversion from layout to depth prediction differentiable, thus making our proposed model end-to-end trainable while leveraging the 3D geometric information, without the need of providing the ground truth depth. Our method achieves state-of-the-art performance on numerous 360 layout benchmark datasets. Moreover, our formulation enables a pre-training step on the depth dataset, which further improves the generalizability of our layout estimation model.

Motivation & Objective

• To address the limitation of 2D pixel-based losses in monocular 360° layout estimation, which fail to capture 3D geometric structure.
• To incorporate 3D geometric supervision into layout estimation without requiring ground-truth depth annotations.
• To enable end-to-end training by making the layout-to-depth transformation differentiable.
• To improve model generalizability through pre-training on synthetic 360° depth datasets.

Proposed method

• The method formulates 360° layout estimation as predicting depth values along the horizon line, termed 'horizon-depth', to preserve 3D geometric consistency.
• It introduces a differentiable L2D (layout-to-depth) procedure based on ray-casting from a unit sphere, enabling gradient flow from depth loss to layout prediction.
• A 'Grid Re-sample' strategy reduces computational cost by approximating the horizon-depth map using a flexible number of rays (default M=256).
• The model leverages standard depth estimation loss functions (e.g., L2 loss) in 3D space, improving geometric accuracy.
• It enables pre-training on synthetic 360° depth datasets such as Structured3D, which provides high-quality depth supervision.
• The framework supports both end-to-end training on layout datasets and transfer learning via depth pre-training.

Experimental results

Research questions

• RQ1 Can differentiable depth rendering improve 3D geometric consistency in monocular 360° layout estimation?
• RQ2 Does formulating layout estimation as a horizon-depth prediction task lead to better performance than conventional 2D pixel-based losses?
• RQ3 Can pre-training on synthetic 360° depth datasets improve generalization across diverse layout distributions?
• RQ4 How sensitive is the model to the number of rays used in the ray-casting process?
• RQ5 Can the proposed method achieve state-of-the-art performance in both within-dataset and cross-dataset evaluation settings?

Key findings

• LED2-Net achieves state-of-the-art performance on Matterport3D, Realtor360, PanoContext, and Stanford2D3D, outperforming HorizonNet and AtlantaNet in both within- and cross-dataset evaluations.
• The model shows significant improvement in cross-dataset generalization when pre-trained on the Structured3D synthetic depth dataset, especially when training on Matterport3D and testing on Realtor360 and Stanford2D3D.
• Pre-training on Structured3D improves performance by around 1% on the Realtor360 within-dataset setting, demonstrating the benefit of 3D geometric priors.
• The model’s performance saturates at M=256 rays in the Grid Re-sample strategy, indicating that higher ray counts provide diminishing returns with increased computational cost.
• The IoU metric shows consistent gains across all datasets, confirming that 3D-aware depth supervision improves layout boundary accuracy.
• The ablation study confirms that the differentiable L2D module is essential for end-to-end training and enables effective use of 3D geometric constraints.