[Paper Review] Pseudocylindrical Convolutions for Learned Omnidirectional Image Compression

Mu Li, Kede Ma+2 more|arXiv (Cornell University)|Dec 25, 2021

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes pseudocylindrical convolutions and a learned pseudocylindrical representation for end-to-end deep neural network-based 360° image compression. By reparameterizing the sphere-to-plane mapping to reduce distortion and enabling efficient convolution via pseudocylindrical padding, the method achieves superior rate-distortion performance and perceptual quality across 19,790 images, outperforming ERP and other projections with up to 25.86% BD-rate savings in VSSIM and 13.97% in BD-entropy.

ABSTRACT

Although equirectangular projection (ERP) is a convenient form to store omnidirectional images (also known as 360-degree images), it is neither equal-area nor conformal, thus not friendly to subsequent visual communication. In the context of image compression, ERP will over-sample and deform things and stuff near the poles, making it difficult for perceptually optimal bit allocation. In conventional 360-degree image compression, techniques such as region-wise packing and tiled representation are introduced to alleviate the over-sampling problem, achieving limited success. In this paper, we make one of the first attempts to learn deep neural networks for omnidirectional image compression. We first describe parametric pseudocylindrical representation as a generalization of common pseudocylindrical map projections. A computationally tractable greedy method is presented to determine the (sub)-optimal configuration of the pseudocylindrical representation in terms of a novel proxy objective for rate-distortion performance. We then propose pseudocylindrical convolutions for 360-degree image compression. Under reasonable constraints on the parametric representation, the pseudocylindrical convolution can be efficiently implemented by standard convolution with the so-called pseudocylindrical padding. To demonstrate the feasibility of our idea, we implement an end-to-end 360-degree image compression system, consisting of the learned pseudocylindrical representation, an analysis transform, a non-uniform quantizer, a synthesis transform, and an entropy model. Experimental results on $19,790$ omnidirectional images show that our method achieves consistently better rate-distortion performance than the competing methods. Moreover, the visual quality by our method is significantly improved for all images at all bitrates.

Motivation & Objective

• To address the non-uniform sampling and distortion issues in equirectangular projection (ERP) for 360° image compression.
• To design a differentiable, learnable map projection that minimizes rate-distortion loss while maintaining computational efficiency.
• To develop pseudocylindrical convolutions that enable standard CNNs to operate directly on non-rectilinear 360° representations.
• To build an end-to-end deep learning compression system using the proposed representation and layers, optimized for perceptual quality and bit-rate efficiency.
• To demonstrate that learned pseudocylindrical representations and convolutions outperform conventional ERP and tiled projections in both rate-distortion and visual quality.

Proposed method

• Proposes a parametric pseudocylindrical representation as a generalization of map projections, with latitude-dependent scaling to balance area and angular distortion.
• Introduces a greedy algorithm to optimize the representation parameters using a proxy rate-distortion objective, minimizing distortion while controlling bit-rate.
• Designs pseudocylindrical convolutions using a novel 'pseudocylindrical padding' technique that enables standard convolution operations on the transformed 360° image grid.
• Employs a standard end-to-end compression framework: analysis transform, non-uniform quantization, synthesis transform, and entropy model, all operating on the learned pseudocylindrical representation.
• Uses a VMSE-based distortion measure and joint optimization of the entire system for rate-distortion performance.
• Validates the method on a large-scale dataset of 19,790 360° images, comparing against ERP, sinusoidal tiles, and tiled representations.

Experimental results

Research questions

• RQ1 Can a differentiable, learnable pseudocylindrical map projection reduce distortion and improve compression efficiency compared to standard ERP?
• RQ2 Does the proposed pseudocylindrical convolution enable efficient and effective feature learning on 360° images without sacrificing computational speed?
• RQ3 Can joint optimization of the pseudocylindrical representation and deep compression network yield better rate-distortion and perceptual performance than fixed projections?
• RQ4 Is the proposed method scalable and transferable to other 360° vision tasks beyond compression?
• RQ5 How does the learned representation compare to sinusoidal or tiled projections in terms of bit-rate savings and visual quality?

Key findings

• The proposed optimized pseudocylindrical representation achieved a 13.97% BD-entropy gain and 25.86% BD-VSSIM gain over baseline methods, indicating superior rate-distortion and perceptual performance.
• The method outperformed both ERP and sinusoidal tiles, with the optimized representation yielding 4.30% bitrate savings and 14.22% PSNR gain over ERP in BD-BR metrics.
• Pseudocylindrical convolutions matched the speed of standard convolutions, with only 0.003s additional latency on average, proving computational feasibility.
• The ablation study confirmed that the learned representation and pseudocylindrical convolution are both essential, with the former contributing more to perceptual gains.
• Visual quality was significantly improved at all bitrates, with consistent gains in PSNR and VSSIM across all test images.
• The method demonstrated strong potential for integration with existing video codecs and broader 360° vision applications due to its compatibility and efficiency.