[Paper Review] Variable Rate Image Compression with Recurrent Neural Networks

George Toderici, Sean M. O’Malley+6 more|arXiv (Cornell University)|Nov 19, 2015

Advanced Data Compression Techniques17 references118 citations

TL;DR

This paper proposes a variable-rate image compression framework using convolutional and deconvolutional LSTM recurrent networks that achieve superior perceptual quality over JPEG, WebP, and JPEG2000 at lower bitrates. The model enables progressive, rate-adaptive compression with a single trained network, outperforming standard codecs on 32×32 thumbnails in SSIM and visual quality while reducing storage by 10% or more.

ABSTRACT

A large fraction of Internet traffic is now driven by requests from mobile devices with relatively small screens and often stringent bandwidth requirements. Due to these factors, it has become the norm for modern graphics-heavy websites to transmit low-resolution, low-bytecount image previews (thumbnails) as part of the initial page load process to improve apparent page responsiveness. Increasing thumbnail compression beyond the capabilities of existing codecs is therefore a current research focus, as any byte savings will significantly enhance the experience of mobile device users. Toward this end, we propose a general framework for variable-rate image compression and a novel architecture based on convolutional and deconvolutional LSTM recurrent networks. Our models address the main issues that have prevented autoencoder neural networks from competing with existing image compression algorithms: (1) our networks only need to be trained once (not per-image), regardless of input image dimensions and the desired compression rate; (2) our networks are progressive, meaning that the more bits are sent, the more accurate the image reconstruction; and (3) the proposed architecture is at least as efficient as a standard purpose-trained autoencoder for a given number of bits. On a large-scale benchmark of 32$ imes$32 thumbnails, our LSTM-based approaches provide better visual quality than (headerless) JPEG, JPEG2000 and WebP, with a storage size that is reduced by 10% or more.

Motivation & Objective

• To address the limitations of autoencoder-based neural networks in image compression, such as fixed-rate encoding and poor generalization across image scales.
• To develop a single, trainable neural network architecture capable of variable-rate compression across diverse image dimensions and target bitrates.
• To improve perceptual quality and reduce artifacts—especially blockiness and color smearing—common in low-bitrate thumbnail compression.
• To enable progressive reconstruction where higher bitrates yield increasingly accurate image reconstructions.
• To eliminate the need for hand-tuned heuristics or codec-specific preprocessing like chroma subsampling.

Proposed method

• The framework employs a recurrent autoencoder with an encoder-decoder structure using convolutional and deconvolutional LSTM layers to model spatial and temporal dependencies in image patches.
• The encoder compresses input images into a bottleneck representation using stacked convolutional LSTMs, while the decoder reconstructs the image using deconvolutional LSTMs.
• A differentiable quantization layer is applied to the bottleneck to produce discrete bitstream representations, enabling end-to-end training with backpropagation.
• The model supports variable-rate compression by adjusting the number of bits allocated to the bottleneck, enabling progressive refinement of the reconstruction.
• The architecture is trained end-to-end on a large-scale dataset of 21 million 32×32 thumbnails, enabling generalization across diverse image content.
• A dynamic bit allocation strategy is applied in the convolutional variant to prioritize high-activity regions, though it may introduce boundary artifacts.

Experimental results

Research questions

• RQ1 Can a single neural network be trained to support variable-rate image compression across different image sizes and target bitrates without retraining?
• RQ2 How does an LSTM-based autoencoder compare to standard codecs like JPEG and WebP in terms of perceptual quality at low bitrates?
• RQ3 Can recurrent architectures effectively model spatial redundancy and preserve high-frequency details in low-resolution thumbnails?
• RQ4 Does the absence of chroma subsampling in the LSTM model lead to higher quality at comparable bitrates compared to codecs that use it?
• RQ5 Can the model achieve progressive reconstruction where increasing bit allocation improves image fidelity?

Key findings

• The (de)convolutional LSTM model achieves higher SSIM scores than JPEG and WebP across all tested low bitrates, with improvements of 4%–12% in perceptual quality at equivalent or lower bitrates.
• At 0.625 bpp, the (de)convolutional LSTM model matches or exceeds the SSIM of JPEG and WebP at higher bitrates, indicating superior rate-distortion efficiency.
• The model reduces visible block artifacts and color bleeding compared to JPEG and WebP, especially at low bitrates, while avoiding excessive smoothing.
• Storage size is reduced by 10% or more compared to JPEG, WebP, and JPEG2000 for the same visual quality on 32×32 thumbnails.
• The model maintains high performance without chroma subsampling, unlike JPEG, which incurs a 26% bitrate increase when using 4:4:4 encoding.
• The framework enables progressive reconstruction: higher bit allocations yield better image reconstructions, and a single model supports all target rates.