[Paper Review] Learned Variable-Rate Multi-Frequency Image Compression using Modulated Generalized Octave Convolution

Jianping Lin, Mohammad Akbari+8 more|arXiv (Cornell University)|Sep 25, 2020

Advanced Data Compression Techniques8 references4 citations

TL;DR

This paper proposes a learned, variable-rate image compression framework using modulated generalized octave convolutions (M-GoConv) to separate latent features into high- and low-frequency components, enabling efficient multi-scale representation. By conditioning the network on a Lagrangian parameter λ, the method achieves high-rate-distortion performance across a wide bitrate range using only three models, outperforming VVC in Y-MS-SSIM and matching BPG in YUV-PSNR.

ABSTRACT

In this proposal, we design a learned multi-frequency image compression approach that uses generalized octave convolutions to factorize the latent representations into high-frequency (HF) and low-frequency (LF) components, and the LF components have lower resolution than HF components, which can improve the rate-distortion performance, similar to wavelet transform. Moreover, compared to the original octave convolution, the proposed generalized octave convolution (GoConv) and octave transposed-convolution (GoTConv) with internal activation layers preserve more spatial structure of the information, and enable more effective filtering between the HF and LF components, which further improve the performance. In addition, we develop a variable-rate scheme using the Lagrangian parameter to modulate all the internal feature maps in the auto-encoder, which allows the scheme to achieve the large bitrate range of the JPEG AI with only three models. Experiments show that the proposed scheme achieves much better Y MS-SSIM than VVC. In terms of YUV PSNR, our scheme is very similar to HEVC.

Motivation & Objective

• To develop a deep learning-based image compression method that supports variable bitrate across a wide range without retraining.
• To improve rate-distortion performance by factorizing latent representations into high- and low-frequency components using generalized octave convolutions.
• To enable efficient, end-to-end training with a single Lagrangian parameter λ that modulates internal feature maps for rate control.
• To achieve high performance comparable to state-of-the-art codecs like BPG and VVC while maintaining low encoding/decoding latency.

Proposed method

• The method employs modulated generalized octave convolution (M-GoConv) and transposed-convolution (M-GoTConv) layers that split feature maps into high-frequency (HF) and low-frequency (LF) components, with LF maps at half spatial resolution.
• Internal feature maps are scaled via a learnable scaling network that maps the Lagrangian parameter λ to channel-wise modulation factors, enabling rate control.
• The network uses a shared architecture with a single set of parameters, where λ is used to condition all internal convolutions, allowing variable-rate operation.
• The model is trained using a multi-rate R-D objective that minimizes the weighted sum of rate and distortion, with distortion measured as 0.9×RGB-MSE + 0.1×(1−Y-MS-SSIM).
• Entropy coding is applied to latent representations using learned hyperpriors, with bitstream rates computed via Gaussian and uniform quantization models.
• Three separate models are trained with different λ sets to cover the full bitrate range of the JPEG-AI CfE standard, enabling accurate bitrate control across all target rates.

Experimental results

Research questions

• RQ1 Can a single learned autoencoder architecture achieve variable-rate image compression across a wide bitrate range without retraining or multiple models?
• RQ2 How does the use of generalized octave convolutions with λ-modulated feature scaling improve rate-distortion performance compared to standard convolutional networks?
• RQ3 To what extent can the proposed method outperform VVC and BPG in terms of Y-MS-SSIM and YUV-PSNR under comparable conditions?
• RQ4 Can the Lagrangian parameter λ be effectively used to condition internal layers for precise bitrate control across diverse image content?
• RQ5 What is the trade-off between model complexity, encoding/decoding speed, and compression performance in this multi-frequency, multi-rate framework?

Key findings

• The proposed method achieves up to 5 dB higher Y-MS-SSIM than the VVC Test Model (VTM) at high bitrates, significantly outperforming it in structural similarity.
• In YUV-PSNR, the method achieves performance comparable to BPG across multiple test images, demonstrating strong fidelity.
• The scheme achieves bitrate accuracy within 1% deviation across all target rates by selecting an appropriate λ value, even outside the training set.
• Encoding and decoding times are 15 and 21 seconds respectively for a 1944×1296 image on CPU, and under 10 seconds on GPU, indicating high efficiency.
• Only three models are needed to cover the entire bitrate range of the JPEG-AI CfE standard, demonstrating strong generalization and scalability.
• The use of modulated generalized octave convolutions preserves spatial structure better than standard octave convolutions, contributing to improved reconstruction quality.