[Paper Review] Slimmable Compressive Autoencoders for Practical Neural Image Compression

Fei Yang, Luis Herranz+2 more|arXiv (Cornell University)|Mar 29, 2021

Advanced Data Compression Techniques42 references4 citations

TL;DR

This paper proposes Slimmable Compressive Autoencoders (SlimCAE), a unified neural image compression framework that enables dynamic rate-distortion tradeoffs and variable computational complexity through trainable, width-adjustable encoder and decoder networks. By optimizing distinct rate-distortion objectives for each network width via a novel $λ$-scheduling training strategy, SlimCAE achieves state-of-the-art rate-distortion performance while enabling on-the-fly adaptation of memory, FLOPs, and latency—making it highly practical for resource-constrained devices.

ABSTRACT

Neural image compression leverages deep neural networks to outperform traditional image codecs in rate-distortion performance. However, the resulting models are also heavy, computationally demanding and generally optimized for a single rate, limiting their practical use. Focusing on practical image compression, we propose slimmable compressive autoencoders (SlimCAEs), where rate (R) and distortion (D) are jointly optimized for different capacities. Once trained, encoders and decoders can be executed at different capacities, leading to different rates and complexities. We show that a successful implementation of SlimCAEs requires suitable capacity-specific RD tradeoffs. Our experiments show that SlimCAEs are highly flexible models that provide excellent rate-distortion performance, variable rate, and dynamic adjustment of memory, computational cost and latency, thus addressing the main requirements of practical image compression.

Motivation & Objective

• Address the practical limitations of existing neural image compression models, which are computationally heavy, optimized for a single rate, and lack dynamic adaptability.
• Enable joint optimization of rate and distortion across multiple network capacities (i.e., different widths) to support variable-rate and variable-complexity operation in a single model.
• Overcome the challenge of suboptimal performance in naively trained slimmable networks by introducing a capacity-aware training strategy that assigns distinct rate-distortion objectives to each width.
• Design novel slimmable components, including generalized divisive normalization (GDN) layers and entropy models, to support efficient, scalable inference.
• Demonstrate the feasibility of generating scalable bitstreams from SlimCAE while preserving key advantages of dynamic complexity control.

Proposed method

• Propose a slimmable compressive autoencoder (SlimCAE) framework where encoder and decoder widths can be dynamically adjusted post-training to control rate, memory, and computational cost.
• Introduce a $λ$-scheduling algorithm that alternates between training the full model and adjusting the $λ$ hyperparameter for each network width, ensuring each width optimizes its own rate-distortion tradeoff.
• Design slimmable GDN layers that maintain nonlinearity and compression efficiency across varying channel widths, enabling smooth capacity scaling.
• Develop slimmable entropy models using conditional convolutions and autoregressive context modeling, allowing variable-rate entropy coding with minimal performance loss.
• Train the model end-to-end using a hybrid loss combining distortion (MSE or MS-SSIM) and rate (cross-entropy), with width-specific $λ$ values to balance tradeoffs.
• Enable scalable bitstream generation by independently encoding groups of channels, supporting base + enhancement stream decoding while maintaining compatibility with the main model.

Experimental results

Research questions

• RQ1 Can a single neural image compression model dynamically adjust its rate, memory footprint, and computational cost without retraining?
• RQ2 How can slimmable neural networks be effectively trained for compressive autoencoders when each network width requires a different rate-distortion tradeoff?
• RQ3 What is the impact of capacity-specific optimization on rate-distortion performance compared to shared optimization across widths?
• RQ4 Can slimmable components (GDN, entropy models) be designed to maintain high compression efficiency across multiple widths?
• RQ5 To what extent can SlimCAE generate scalable bitstreams while preserving dynamic complexity control and rate-distortion performance?

Key findings

• SlimCAE achieves state-of-the-art rate-distortion performance on Kodak and Tecnick datasets, with BD-rate gains of up to -47.88% over BPG when optimizing for MS-SSIM.
• The model's memory footprint ranges from 27.1 MB to 62.5 MB depending on width, significantly lower than multiple CAEs (42.9–78.3 MB) and comparable to single models.
• SlimCAE reduces FLOPs and latency dynamically by adjusting network width, enabling efficient deployment on resource-constrained devices.
• The $λ$-scheduling training method enables each width to achieve its optimal rate-distortion tradeoff, outperforming naive joint training by up to 9.52 dB in PSNR on Kodak.
• Slimmable entropy models introduce negligible performance loss (e.g., -6.17% BD-rate) while enabling variable-rate and scalable bitstream generation.
• The framework supports both dynamic complexity scaling and scalable bitstream generation, making it suitable for heterogeneous networks and adaptive streaming.