[Paper Review] Surface Normal Estimation of Tilted Images via Spatial Rectifier

Tien Do, Khiem Vuong+2 more|arXiv (Cornell University)|Jul 17, 2020

Advanced Vision and Imaging41 references4 citations

TL;DR

This paper proposes a spatial rectifier network that learns to warp tilted RGB images into a rectified space where surface normal estimation performance matches that of gravity-aligned training data. By jointly training the rectifier and a normal estimator using synthetic tilted images, and introducing a truncated angular loss for improved generalization, the method achieves state-of-the-art accuracy on ScanNet, NYUv2, and a new Tilt-RGBD dataset with large camera roll and pitch, while reducing FLOPS by 70% compared to prior models.

ABSTRACT

In this paper, we present a spatial rectifier to estimate surface normals of tilted images. Tilted images are of particular interest as more visual data are captured by arbitrarily oriented sensors such as body-/robot-mounted cameras. Existing approaches exhibit bounded performance on predicting surface normals because they were trained using gravity-aligned images. Our two main hypotheses are: (1) visual scene layout is indicative of the gravity direction; and (2) not all surfaces are equally represented by a learned estimator due to the structured distribution of the training data, thus, there exists a transformation for each tilted image that is more responsive to the learned estimator than others. We design a spatial rectifier that is learned to transform the surface normal distribution of a tilted image to the rectified one that matches the gravity-aligned training data distribution. Along with the spatial rectifier, we propose a novel truncated angular loss that offers a stronger gradient at smaller angular errors and robustness to outliers. The resulting estimator outperforms the state-of-the-art methods including data augmentation baselines not only on ScanNet and NYUv2 but also on a new dataset called Tilt-RGBD that includes considerable roll and pitch camera motion.

Motivation & Objective

• To address the performance drop in surface normal estimation when testing on tilted images due to domain shift from gravity-aligned training data.
• To learn a global geometric transformation (spatial rectifier) that aligns the surface normal distribution of tilted images with that of upright training data.
• To improve training convergence and robustness by designing a truncated angular loss that better matches the evaluation metric and enhances performance on small angular errors.
• To reduce computational cost while maintaining high accuracy through a novel network architecture with dilated convolutions and larger receptive fields.

Proposed method

• The spatial rectifier is parameterized by the estimated gravity direction and a principal direction of the tilted image, learning a homography-like transformation to align the surface normal distribution of the input image with that of the training data.
• The rectifier is trained end-to-end via synthetic data generation: a family of homographies induced by gravity and principal directions is used to simulate tilted images from the ScanNet dataset.
• A truncated angular loss (TAL) is introduced, which provides stronger gradients at small angular errors (below 7.5°) and is robust to outliers, addressing the mismatch between L2 loss and angular evaluation metrics.
• The network architecture incorporates dilated convolutions in decoding layers to increase receptive fields without increasing FLOPS, enabling larger models like ResNeXt-101 with reduced computational cost.
• The method is evaluated on ScanNet, NYUv2, and a new Tilt-RGBD dataset with significant roll and pitch motion, demonstrating robustness across diverse camera orientations.
• The full pipeline is trained jointly: the spatial rectifier and normal estimator are optimized together using synthetic tilted images, enabling domain-invariant feature learning.

Experimental results

Research questions

• RQ1 Can a learned spatial transformation effectively reduce the domain gap between tilted test images and gravity-aligned training data for surface normal estimation?
• RQ2 Does a truncated angular loss that emphasizes small angular errors improve model generalization and performance on standard evaluation metrics?
• RQ3 Can dilated convolutions in the decoder path increase receptive field and model expressiveness without increasing FLOPS or memory consumption?
• RQ4 How does the proposed method compare to state-of-the-art approaches, including data augmentation baselines, on datasets with high camera tilt (e.g., Tilt-RGBD)?
• RQ5 Can the spatial rectifier generalize across diverse camera orientations, including large roll and pitch angles, without requiring explicit supervision on tilt angles?

Key findings

• The proposed method achieves state-of-the-art performance on ScanNet and NYUv2, with a mean angular error of 15.0° and 16.2° respectively, outperforming FrameNet and VPLNet.
• On the new Tilt-RGBD dataset with large roll and pitch, the method achieves 80.0% and 85.2% accuracy at 11.25° and 22.5° thresholds on ScanNet, and 77.1% at 22.5° on NYUv2, demonstrating robustness to tilt.
• The truncated angular loss (TAL) improves performance at tight thresholds (e.g., 5° and 7.5°), achieving 50.2% and 34.2% of pixels within 7.5° and 5° error on NYUv2, respectively, outperforming L2 and angular loss.
• The DFPN+TAL+SR model with ResNeXt-101 backbone reduces FLOPS by 70% compared to DORN, achieving 30 FPS inference on a GTX 1660 while maintaining high accuracy.
• The spatial rectifier reduces estimation error in underrepresented normal regions (e.g., d, e, f in Figure 1) by transforming them to densely represented regions (d', e', f'), significantly improving prediction quality on tilted images.
• The method outperforms data augmentation baselines and SOTA models across all metrics, including 69.3% and 83.9% accuracy at 30° threshold on ScanNet and NYUv2, respectively, with lower FLOPS and memory usage.