[Paper Review] SpherePHD: Applying CNNs on a Spherical PolyHeDron Representation of 360 degree Images

Yeonkun Lee, Jaeseok Jeong+3 more|arXiv (Cornell University)|Nov 20, 2018

Advanced Vision and Imaging22 references10 citations

TL;DR

This paper proposes SpherePHD, a novel spherical polyhedron representation of 360° images using an icosahedral geodesic mesh to minimize spatial distortion and preserve continuity. By redefining convolution and pooling operations on this geometry, the method reduces non-uniform spatial resolving power and edge discontinuities inherent in ERP and cube map projections, achieving superior performance in classification, detection, and semantic segmentation on synthetic and real-world datasets, especially under camera tilt and data augmentation.

ABSTRACT

Omni-directional cameras have many advantages overconventional cameras in that they have a much wider field-of-view (FOV). Accordingly, several approaches have beenproposed recently to apply convolutional neural networks(CNNs) to omni-directional images for various visual tasks.However, most of them use image representations defined inthe Euclidean space after transforming the omni-directionalviews originally formed in the non-Euclidean space. Thistransformation leads to shape distortion due to nonuniformspatial resolving power and the loss of continuity. Theseeffects make existing convolution kernels experience diffi-culties in extracting meaningful information.This paper presents a novel method to resolve such prob-lems of applying CNNs to omni-directional images. Theproposed method utilizes a spherical polyhedron to rep-resent omni-directional views. This method minimizes thevariance of the spatial resolving power on the sphere sur-face, and includes new convolution and pooling methodsfor the proposed representation. The proposed method canalso be adopted by any existing CNN-based methods. Thefeasibility of the proposed method is demonstrated throughclassification, detection, and semantic segmentation taskswith synthetic and real datasets.

Motivation & Objective

• To address spatial distortion and discontinuity in CNNs applied to 360° images due to non-Euclidean to Euclidean space transformations.
• To reduce variance in spatial resolving power caused by equi-rectangular projection (ERP) and cube map representations.
• To preserve the cyclical and continuous nature of omni-directional images during CNN processing.
• To develop geometry-aware convolution and pooling operations compatible with existing CNN architectures on the new representation.
• To demonstrate improved performance across classification, object detection, and semantic segmentation tasks on both synthetic and real datasets.

Proposed method

• The paper introduces SpherePHD, a spherical polyhedron representation derived from an icosahedral geodesic mesh to minimize spatial distortion and variance in spatial resolving power.
• It proposes a novel projection method that maps 360° images onto the faces of the icosahedron, preserving geometric continuity and reducing shape distortion compared to ERP and cube maps.
• The method defines a new convolution operation on the spherical polyhedron by leveraging local neighborhoods on the mesh, ensuring consistent receptive field shapes across the sphere.
• A custom pooling layer is designed to downsample features while preserving spatial structure, using max-pooling with index tracking for unpooling in autoencoder architectures.
• The framework is compatible with existing CNNs, allowing plug-in integration of SpherePHD into standard networks without architectural overhaul.
• The implementation supports data augmentation via random camera tilting, enhancing robustness to real-world variations.

Experimental results

Research questions

• RQ1 Can a spherical polyhedron representation reduce spatial distortion and improve spatial resolving power uniformity in 360° image processing compared to ERP and cube map projections?
• RQ2 How does the proposed geometry-aware convolution and pooling operation perform in maintaining feature consistency across the sphere compared to standard CNNs on distorted projections?
• RQ3 Does the SpherePHD representation improve detection and segmentation accuracy under camera tilt and data augmentation, where ERP and cube maps fail?
• RQ4 To what extent does the preservation of cyclical continuity in SpherePHD reduce object split artifacts in detection tasks?
• RQ5 Can the SpherePHD framework be effectively integrated into existing CNN-based models without requiring major architectural changes?

Key findings

• On the SYNTHIA dataset, SpherePHD achieved 64.52% average precision in vehicle detection under rotation-augmented data, significantly outperforming ERP (39.87%) and cube map (26.03%) representations.
• For semantic segmentation on the SYNTHIA dataset, SpherePHD achieved 70.08% average class accuracy and 97.20% overall pixel accuracy, surpassing ERP (62.69% and 95.07%) and cube map (36.07% and 66.04%) methods.
• On the more challenging Stanford2D3D real-world dataset, SpherePHD achieved 26.40% average class accuracy and 51.40% overall pixel accuracy, substantially outperforming ERP (17.97% and 35.02%) and cube map (17.42% and 32.38%) representations.
• The performance of ERP representation degraded under data augmentation, while SpherePHD maintained high accuracy, indicating its robustness to camera tilt and spatial variation.
• Visual results in Fig. 12 and Fig. 13 confirmed that SpherePHD reduced object split artifacts and improved segmentation consistency, especially near image borders and on tilted images.
• The proposed method demonstrated consistent superiority across all tasks—classification, detection, and segmentation—on both synthetic and real-world datasets, particularly under challenging conditions like camera tilt and high-class diversity.