[Paper Review] SphereSR: 360° Image Super-Resolution with Arbitrary Projection via Continuous Spherical Image Representation

Young-Ho Yoon, Inchul Chung+2 more|arXiv (Cornell University)|Dec 13, 2021

Advanced Vision and Imaging4 citations

TL;DR

SphereSR proposes a novel 360° image super-resolution framework that generates high-resolution omnidirectional images with arbitrary projections using a continuous spherical representation. It employs geometry-aligned convolution for efficient spherical feature extraction and a spherical local implicit image function (SLIIF) to predict RGB values at any spherical coordinate, achieving state-of-the-art performance across multiple benchmarks and projection types.

ABSTRACT

The 360°imaging has recently gained great attention; however, its angular resolution is relatively lower than that of a narrow field-of-view (FOV) perspective image as it is captured by using fisheye lenses with the same sensor size. Therefore, it is beneficial to super-resolve a 360°image. Some attempts have been made but mostly considered the equirectangular projection (ERP) as one of the way for 360°image representation despite of latitude-dependent distortions. In that case, as the output high-resolution(HR) image is always in the same ERP format as the low-resolution (LR) input, another information loss may occur when transforming the HR image to other projection types. In this paper, we propose SphereSR, a novel framework to generate a continuous spherical image representation from an LR 360°image, aiming at predicting the RGB values at given spherical coordinates for super-resolution with an arbitrary 360°image projection. Specifically, we first propose a feature extraction module that represents the spherical data based on icosahedron and efficiently extracts features on the spherical surface. We then propose a spherical local implicit image function (SLIIF) to predict RGB values at the spherical coordinates. As such, SphereSR flexibly reconstructs an HR image under an arbitrary projection type. Experiments on various benchmark datasets show that our method significantly surpasses existing methods.

Motivation & Objective

• Address the limitations of existing 360° image super-resolution methods that are restricted to equirectangular projection (ERP), which introduces angular distortions and limits flexibility.
• Overcome the pixel density variation and distortion issues inherent in 2D planar projections like ERP by modeling 360° images on a spherical surface using an icosahedral mesh.
• Enable high-resolution image reconstruction for any projection type (e.g., cube map, equirectangular) by predicting RGB values at arbitrary spherical coordinates.
• Develop an efficient and memory-optimized data structure and network module to support spherical deep learning without relying on planar image transformations.

Proposed method

• Propose a geometry-aligned convolution (GA-Conv) layer that operates directly on an icosahedral mesh to extract features while preserving spherical geometry and minimizing distortion.
• Introduce a spherical local implicit image function (SLIIF) that predicts RGB values at any spherical coordinate using position embeddings based on normal plane polar coordinates on triangular faces.
• Use a continuous spherical representation to avoid pixel misalignment during projection transformations, enabling flexible output in any 360° projection format.
• Implement a feature loss that measures similarity between feature maps of different projection types, improving generalization and performance across diverse projections.
• Design a memory-efficient data structure for spherical feature storage and processing, reducing activation memory compared to prior methods like SpherePHD.
• Train the model end-to-end using a combination of L1 loss and the proposed feature loss to enhance perceptual quality and feature consistency across projections.

Experimental results

Research questions

• RQ1 Can a deep learning framework achieve 360° image super-resolution with arbitrary projection outputs, rather than being limited to equirectangular projection?
• RQ2 How can spherical image representation be made continuous and differentiable to enable high-fidelity RGB prediction at arbitrary spherical coordinates?
• RQ3 What is the impact of using an icosahedral mesh with geometry-aligned convolutions on feature extraction efficiency and distortion reduction compared to planar projections?
• RQ4 To what extent does a projection-agnostic feature loss improve performance across different 360° image projections?
• RQ5 How does the proposed data structure compare to existing spherical CNN implementations in terms of memory efficiency and scalability?

Key findings

• SphereSR achieves state-of-the-art performance on both ODI-SR and SUN 360 Panorama datasets, with a WS-PSNR of 24.37 dB and WS-SSIM of 0.6777 at ×8 super-resolution.
• The inclusion of GA-Conv improves PSNR by 0.14 dB on ODI-SR and 0.12 dB on SUN 360 Panorama at ×8 SR, demonstrating its effectiveness in feature extraction.
• SLIIF outperforms the pixel-shuffle baseline by 0.03 dB in WS-PSNR at ×8 SR, confirming its advantage in continuous, coordinate-based image reconstruction.
• The feature loss leads to consistent performance gains across all scales and datasets, with improvements of 0.04–0.05 dB in WS-PSNR and 0.001–0.002 in WS-SSIM.
• The proposed data structure reduces activation memory by up to 70% compared to SpherePHD at higher input levels, with the memory efficiency ratio improving as resolution increases.
• SphereSR enables flexible super-resolution output in any projection type (e.g., ERP, cube map) without re-projection artifacts, as the model predicts RGB values directly on the sphere.