[Paper Review] Surface Normals in the Wild

Weifeng Chen, Donglai Xiang+1 more|arXiv (Cornell University)|Apr 10, 2017

Advanced Vision and Imaging1 references4 citations

TL;DR

This paper introduces the Surface Normals in the Wild (SNOW) dataset, a large-scale collection of crowdsourced surface normal annotations for unconstrained images. It proposes two novel loss functions—angular difference and depth derivative comparison—that leverage surface normals to improve single-image depth estimation, significantly enhancing depth quality on both NYU Depth and SNOW datasets compared to prior methods.

ABSTRACT

We study the problem of single-image depth estimation for images in the wild. We collect human annotated surface normals and use them to train a neural network that directly predicts pixel-wise depth. We propose two novel loss functions for training with surface normal annotations. Experiments on NYU Depth and our own dataset demonstrate that our approach can significantly improve the quality of depth estimation in the wild.

Motivation & Objective

• To address the limitation of relative depth annotations in capturing surface geometry and perceptual fidelity for depth estimation in unconstrained scenes.
• To develop a practical method for crowdsourcing surface normal annotations for diverse, real-world images.
• To train a deep neural network for metric depth prediction using surface normal annotations as supervisory signals.
• To evaluate the effectiveness of surface normal supervision in improving depth estimation quality on both real and synthetic datasets.

Proposed method

• Designed a user interface allowing annotators to adjust a virtual arrow and tangent plane to estimate surface normals, enabling reliable crowdsourcing of surface normals for real-world images.
• Collected 60,061 surface normal annotations from Flickr images, forming the SNOW dataset for training and evaluation.
• Proposed a novel angular difference loss that minimizes the angle between predicted and ground-truth surface normals to improve geometric fidelity.
• Developed a depth-derivative-based loss that compares the derivatives of predicted depth with those derived from ground-truth normals to enhance depth discontinuity accuracy.
• Trained a deep convolutional network using a combination of surface normal supervision and standard depth supervision, with ablation studies to evaluate trade-offs.
• Fine-tuned models on the SNOW dataset to adapt to diverse, unconstrained scenes, demonstrating improved generalization and surface normal accuracy.

Experimental results

Research questions

• RQ1 Can surface normal annotations from human crowdsourcing improve depth estimation in unconstrained, real-world images?
• RQ2 How do surface normal annotations compare to relative depth annotations in capturing perceptually important 3D geometry such as continuity and curvature?
• RQ3 What are the relative advantages of angular difference vs. depth derivative-based losses for surface normal supervision in depth estimation?
• RQ4 Can surface normal supervision significantly improve depth prediction performance when ground-truth depth is unavailable?
• RQ5 To what extent does fine-tuning on the SNOW dataset improve generalization to images in the wild?

Key findings

• The proposed surface normal-based loss functions significantly improve depth estimation quality on the NYU Depth dataset, with the best model (d_n_al_F) outperforming state-of-the-art methods like Wang [31] in angle distance and percentage within threshold metrics.
• Fine-tuning the d_n_al_F model on the SNOW dataset (d_n_al_F_SNOW) led to a 25.75° mean angular error, a 21.66% improvement in the percentage of normals within 11.25°, and a 67.88% improvement within 30°, demonstrating strong generalization to real-world scenes.
• The model trained on SNOW alone (Ours_NYU_SNOW§) achieved a 23.33° mean angular error and 72.74% of normals within 30°, outperforming state-of-the-art surface normal estimation models on the SNOW benchmark.
• The depth-derivative-based loss was found to be more effective for preserving depth discontinuities, while the angular loss better preserved surface orientation fidelity, indicating distinct trade-offs for different applications.
• The SNOW dataset enables effective transfer learning, as models fine-tuned on SNOW show substantial improvements in surface normal prediction accuracy over baseline models.
• Qualitative results in Figure 7 demonstrate that the proposed method produces more perceptually faithful depth predictions on diverse, unconstrained images in the wild.