[Paper Review] Labeling Panoramas with Spherical Hourglass Networks

Carlos Esteves, Kostas Daniilidis+1 more|arXiv (Cornell University)|Sep 6, 2018

Advanced Vision and Imaging2 references3 citations

TL;DR

This paper introduces the Spherical Hourglass Network (SCHN), a rotation-equivariant deep learning model for dense semantic labeling on spherical panoramas using spherical convolutions in the spectral domain. By leveraging SO(3)-equivariant residual bottleneck blocks in an hourglass architecture, SCHN achieves state-of-the-art performance on spherical semantic segmentation without requiring upright image orientation, outperforming standard 2D CNNs under arbitrary camera rotations.

ABSTRACT

With the recent proliferation of consumer-grade 360° cameras, it is worth revisiting visual perception challenges with spherical cameras given the potential benefit of their global field of view. To this end we introduce a spherical convolutional hourglass network (SCHN) for the dense labeling on the sphere. The SCHN is invariant to camera orientation (lifting the usual requirement for `upright' panoramic images), and its design is scalable for larger practical datasets. Initial experiments show promising results on a spherical semantic segmentation task.

Motivation & Objective

• To address the challenge of dense pixel-level labeling on spherical panoramic images, which are commonly distorted when projected onto a 2D plane.
• To develop a deep learning model that is invariant to camera orientation, eliminating the need for data augmentation over all 3D rotations.
• To design a scalable, efficient architecture for spherical semantic segmentation that maintains rotational equivariance while supporting high-resolution output.
• To overcome limitations of standard 2D CNNs on equirectangular projections, which introduce artificial distortions due to spherical topology.

Proposed method

• The model uses spherical convolutional layers based on spherical harmonics and spectral domain operations, ensuring SO(3)-equivariance.
• It employs spherical residual bottleneck blocks with 1×1, 3×3, and 1×1 convolutions to reduce computational cost while preserving performance.
• The architecture follows an hourglass design with downsampling and upsampling blocks to generate high-resolution dense predictions.
• Filter localization is achieved via a small number of spherical Fourier coefficient anchors, enabling localized, smooth, and deformable receptive fields.
• The network is trained using a supervised loss on labeled spherical panoramas, with no need for orientation-specific data augmentation.
• A 2D baseline (2DHG) is used for comparison, replacing spherical convolutions with standard 2D convolutions on equirectangular projections.

Experimental results

Research questions

• RQ1 Can a deep learning model achieve rotation-equivariant semantic segmentation on spherical panoramas without relying on data augmentation for all 3D rotations?
• RQ2 How does the performance of a spherical CNN-based hourglass network compare to standard 2D CNNs on equirectangular projections in terms of generalization under arbitrary camera orientations?
• RQ3 To what extent does using localized spherical filters improve segmentation accuracy compared to global filters in spherical networks?
• RQ4 Can the spherical residual bottleneck block design maintain high performance while reducing computational cost in dense labeling tasks on the sphere?

Key findings

• SCHN achieves a mean Intersection over Union (mIoU) of 0.5582 on test data with arbitrary camera orientations, outperforming the 2D baseline (2DHG) which scores 0.5292 under the same conditions.
• Under canonical orientation, SCHN achieves 0.5683 mIoU, demonstrating strong performance even when compared to models trained under ideal conditions.
• The model generalizes well to unseen orientations, with a mIoU of 0.5024 when training on canonical and testing on arbitrary orientations, indicating robustness to rotation.
• Using larger model variants (SCHN/large) improves performance, achieving 0.5983 mIoU on canonical test data, showing scalability.
• Localizing filters via a small number of spherical Fourier anchors improves performance over global filters, with a 0.5343 mIoU for SCHN/global versus 0.5376 for SCHN with localized filters.
• The proposed architecture outperforms prior work such as Im2Pano3D on the same benchmark, which achieved only 0.330 mIoU on a harder extrapolation task.