[Paper Review] Computationally-Efficient Neural Image Compression with Shallow Decoders

Yibo Yang, Stephan Mandt|arXiv (Cornell University)|Apr 13, 2023

Cell Image Analysis Techniques4 citations

TL;DR

This paper proposes a computationally efficient neural image compression method using a shallow or linear synthesis (decoding) transform inspired by JPEG, achieving state-of-the-art rate-distortion performance at under 50K FLOPs/pixel for decoding—reducing decoding complexity by over 80% compared to standard baselines—by leveraging powerful iterative encoding and a strong hyperprior.

ABSTRACT

Neural image compression methods have seen increasingly strong performance in recent years. However, they suffer orders of magnitude higher computational complexity compared to traditional codecs, which hinders their real-world deployment. This paper takes a step forward towards closing this gap in decoding complexity by using a shallow or even linear decoding transform resembling that of JPEG. To compensate for the resulting drop in compression performance, we exploit the often asymmetrical computation budget between encoding and decoding, by adopting more powerful encoder networks and iterative encoding. We theoretically formalize the intuition behind, and our experimental results establish a new frontier in the trade-off between rate-distortion and decoding complexity for neural image compression. Specifically, we achieve rate-distortion performance competitive with the established mean-scale hyperprior architecture of Minnen et al. (2018) at less than 50K decoding FLOPs/pixel, reducing the baseline's overall decoding complexity by 80%, or over 90% for the synthesis transform alone. Our code can be found at https://github.com/mandt-lab/shallow-ntc.

Motivation & Objective

• Address the high decoding complexity of neural image compression, which hinders real-world deployment despite strong rate-distortion performance.
• Reduce the computational burden of the synthesis (decoding) transform, which accounts for over 80% of decoding complexity in existing methods.
• Explore the feasibility of using lightweight, linear, or shallow decoding transforms similar to classical codecs like JPEG, while preserving high compression quality.
• Leverage the asymmetry in computation budgets between encoding and decoding by using powerful encoders to compensate for simple decoders.
• Establish a new frontier in the trade-off between rate-distortion performance and decoding complexity for neural image compression.

Proposed method

• Replace the deep convolutional synthesis transform in standard neural image compression with a shallow or linear transform, emulating the structure of JPEG.
• Use the Mean-Scale Hyperprior architecture from Minnen et al. (2018) as the base model, preserving the hyperprior but simplifying the synthesis path.
• Employ iterative inference via Stochastic Gumbel Annealing (SGA) to improve the discrete latent representation, enhancing rate-distortion performance.
• Introduce a single hidden layer in the synthesis path to restore expressive power lost by using a shallow or linear transform.
• Formalize the R-D cost in an asymptotic setting to theoretically justify the use of powerful encoders with lightweight decoders.
• Measure decoding complexity in FLOPs per pixel and KMACs (thousand multiply-accumulate operations), enabling direct comparison with classical codecs.

Experimental results

Research questions

• RQ1 Can a shallow or linear synthesis transform achieve competitive rate-distortion performance compared to deep neural decoders in neural image compression?
• RQ2 To what extent can powerful encoding methods, such as iterative inference, compensate for reduced decoding complexity?
• RQ3 How does the curvature and linearity of the latent manifold induced by the synthesis transform affect compression performance?
• RQ4 What is the theoretical relationship between encoding complexity, decoding complexity, and rate-distortion cost in neural lossy compression?
• RQ5 Can a JPEG-like synthesis transform outperform deeper architectures when paired with advanced encoding strategies?

Key findings

• The proposed method achieves rate-distortion performance competitive with the established Mean-Scale Hyperprior architecture at less than 50K FLOPs/pixel for decoding, reducing baseline decoding complexity by over 80%.
• The synthesis transform alone is reduced by more than 90% in complexity, demonstrating the feasibility of near-classical-complexity decoding.
• Despite the simplicity of the synthesis transform, the method maintains strong R-D performance due to the use of iterative encoding and a powerful hyperprior.
• The latent manifold induced by the synthesis transform is found to be relatively flat and preserves linear combinations, suggesting that nonlinearity is not essential for high R-D performance in this regime.
• Theoretical analysis confirms that increasing encoding complexity reduces the inference gap and improves R-D cost, justifying the use of powerful encoders with lightweight decoders.
• Entropy decoding now accounts for 50%–80% of total decoding computation, indicating a shift in computational bottleneck that opens new optimization opportunities.