[Paper Review] Variable-Rate Deep Image Compression through Spatially-Adaptive Feature Transform

Myungseo Song, Jin-Young Choi+1 more|arXiv (Cornell University)|Aug 21, 2021

Advanced Image Processing Techniques42 references4 citations

TL;DR

This paper proposes a single deep neural network for variable-rate image compression that uses a spatially-adaptive feature transform (SFT) to condition compression on a continuous, pixel-wise quality map. The model enables task-aware compression by optimizing quality maps via backpropagation without retraining, achieving superior rate-distortion performance and improved task-specific accuracy—e.g., classification and text preservation—compared to fixed-rate models.

ABSTRACT

We propose a versatile deep image compression network based on Spatial Feature Transform (SFT arXiv:1804.02815), which takes a source image and a corresponding quality map as inputs and produce a compressed image with variable rates. Our model covers a wide range of compression rates using a single model, which is controlled by arbitrary pixel-wise quality maps. In addition, the proposed framework allows us to perform task-aware image compressions for various tasks, e.g., classification, by efficiently estimating optimized quality maps specific to target tasks for our encoding network. This is even possible with a pretrained network without learning separate models for individual tasks. Our algorithm achieves outstanding rate-distortion trade-off compared to the approaches based on multiple models that are optimized separately for several different target rates. At the same level of compression, the proposed approach successfully improves performance on image classification and text region quality preservation via task-aware quality map estimation without additional model training. The code is available at the project website: https://github.com/micmic123/QmapCompression

Motivation & Objective

• Address the limitation of fixed-rate deep compression models that require separate training for each target bitrate.
• Enable spatially-adaptive compression where different image regions are compressed at different rates based on pixel-wise importance.
• Develop a method to generate task-specific quality maps at inference time without fine-tuning the main compression model.
• Achieve state-of-the-art rate-distortion performance across a wide range of bitrates using a single model.
• Preserve critical image content for downstream tasks (e.g., classification, text recognition) by optimizing quality maps for task-specific objectives.

Proposed method

• Condition the image compression network on a real-valued, continuous quality map that specifies pixel-wise importance for distortion weighting.
• Employ spatially-adaptive feature transform (SFT) layers to modulate feature maps based on the quality map, enabling per-pixel rate adaptation.
• Train the network using a rate-distortion loss that incorporates the quality map as a spatially-varying weight for mean squared error (MSE) distortion.
• Optimize task-aware quality maps at inference time by minimizing a rate-task loss using backpropagation, leveraging a pretrained task model (e.g., VGG16) for guidance.
• Use a single end-to-end trainable model to cover a wide range of compression rates, avoiding the need for multiple independently trained models.
• Integrate the quality map optimization into the encoding phase without fine-tuning the main compression network, enabling fast, on-the-fly adaptation.

Experimental results

Research questions

• RQ1 Can a single deep compression model achieve high performance across a wide range of compression rates using a continuous, pixel-wise quality map?
• RQ2 Can task-aware quality maps be generated at test time without retraining the compression model, and does this improve downstream task performance?
• RQ3 How does the proposed method compare to multiple fixed-rate models in terms of rate-distortion performance and task-specific accuracy?
• RQ4 To what extent does spatial adaptivity via SFT improve reconstruction quality in regions of interest compared to uniform or patchwise compression?
• RQ5 Does the absence of an autoregressive context model in the proposed framework limit performance, and can it be enhanced with such components?

Key findings

• The proposed model outperforms single-rate models like M&S and M&S+Context in rate-distortion performance, even without using autoregressive context models.
• The model achieves better or comparable performance to state-of-the-art methods, including those with complex context modeling, while being more flexible and efficient.
• Task-aware quality map optimization at test time improves image classification accuracy by up to 12.5% compared to uniform quality maps, with less than one second inference cost on a TitanXp GPU.
• When guided by Grad-CAM or manually defined text ROIs, the model preserves text regions better than uniform compression, even at low bitrates.
• The ablation study confirms that source conditioning (using the original image as input to the quality map generator) significantly improves performance, especially in spatial adaptivity.
• The model maintains high MS-SSIM and PSNR scores across diverse quality maps, with reconstructed images showing superior texture preservation compared to BPG and M&S Hyperprior.