[Paper Review] Deep Surface Normal Estimation with Hierarchical RGB-D Fusion

Jin Zeng, Yanfeng Tong+5 more|arXiv (Cornell University)|Apr 6, 2019

Advanced Vision and Imaging34 references4 citations

TL;DR

This paper proposes a hierarchical RGB-D fusion network with adaptive feature re-weighting for accurate surface normal estimation from single RGB-D images. By fusing multi-scale features from RGB and depth branches while using a confidence map to suppress noisy depth inputs and a hybrid multi-scale loss to handle imperfect ground-truth data, the method achieves state-of-the-art performance with sharper details and improved robustness to depth corruption.

ABSTRACT

The growing availability of commodity RGB-D cameras has boosted the applications in the field of scene understanding. However, as a fundamental scene understanding task, surface normal estimation from RGB-D data lacks thorough investigation. In this paper, a hierarchical fusion network with adaptive feature re-weighting is proposed for surface normal estimation from a single RGB-D image. Specifically, the features from color image and depth are successively integrated at multiple scales to ensure global surface smoothness while preserving visually salient details. Meanwhile, the depth features are re-weighted with a confidence map estimated from depth before merging into the color branch to avoid artifacts caused by input depth corruption. Additionally, a hybrid multi-scale loss function is designed to learn accurate normal estimation given noisy ground-truth dataset. Extensive experimental results validate the effectiveness of the fusion strategy and the loss design, outperforming state-of-the-art normal estimation schemes.

Motivation & Objective

• To address the lack of effective fusion strategies for combining RGB and depth features in surface normal estimation.
• To reduce artifacts caused by depth sensor noise and missing pixels in depth maps.
• To improve normal estimation accuracy despite noisy or incomplete ground-truth normal maps from multiview reconstruction.
• To design a loss function that balances global smoothness and local detail preservation under imperfect supervision.

Proposed method

• A hierarchical fusion mechanism integrates features from RGB and depth branches at multiple decoder scales to preserve both global smoothness and local sharpness.
• A confidence map is estimated from the input depth to re-weight depth features before fusion, reducing influence from low-confidence regions like edges and holes.
• The network uses a multi-branch encoder-decoder architecture with skip connections to maintain spatial resolution and feature fidelity.
• A hybrid multi-scale loss combines L1 and L2 losses at different resolutions: L1 for fine details at high resolution and L2 for coarse accuracy at low resolution.
• The confidence map is computed via a lightweight subnetwork to assess depth reliability, especially in regions with high uncertainty.
• The model is trained on Matterport3D and ScanNet datasets using ground-truth normals from multiview reconstruction, with loss function designed to mitigate reconstruction noise.

Experimental results

Research questions

• RQ1 How can RGB and depth features be effectively fused across multiple scales to improve surface normal estimation?
• RQ2 Can confidence-based re-weighting of depth features reduce artifacts from depth sensor noise and missing data?
• RQ3 Does a hybrid multi-scale loss function improve performance when training on noisy ground-truth normal maps?
• RQ4 How does hierarchical fusion compare to early or late fusion strategies in terms of detail preservation and accuracy?

Key findings

• The proposed hierarchical fusion with confidence re-weighting outperforms early and late fusion baselines, especially in depth hole regions, as shown in qualitative comparisons.
• On the Matterport3D dataset, the method achieves a mean error of 13.062° and a median error of 6.090° at 30° threshold, surpassing state-of-the-art methods.
• On the ScanNet dataset, the method achieves a mean error of 14.590° and a median error of 7.468° at 30° threshold, demonstrating strong generalization across datasets.
• The ablation study confirms that the hybrid loss function yields sharper results with better detail preservation than L1 or L2 loss alone.
• The confidence map significantly improves performance near depth holes, reducing boundary artifacts compared to using a binary mask.
• The model achieves state-of-the-art performance with a reasonable inference time of 34.2 ms per image on a GTX TITAN X GPU.