[Paper Review] End-to-end Optimized Image Compression

Johannes Ballé, Valero Laparra+1 more|arXiv (Cornell University)|Nov 5, 2016

Image and Signal Denoising Methods31 references1,010 citations

TL;DR

The paper trains a nonlinear transform coding model with a generalized divisive normalization nonlinearity and uniform quantization, optimized end-to-end for rate–distortion, achieving superior perceptual quality and competitive rate–distortion versus JPEG and JPEG 2000.

ABSTRACT

We describe an image compression method, consisting of a nonlinear analysis transformation, a uniform quantizer, and a nonlinear synthesis transformation. The transforms are constructed in three successive stages of convolutional linear filters and nonlinear activation functions. Unlike most convolutional neural networks, the joint nonlinearity is chosen to implement a form of local gain control, inspired by those used to model biological neurons. Using a variant of stochastic gradient descent, we jointly optimize the entire model for rate-distortion performance over a database of training images, introducing a continuous proxy for the discontinuous loss function arising from the quantizer. Under certain conditions, the relaxed loss function may be interpreted as the log likelihood of a generative model, as implemented by a variational autoencoder. Unlike these models, however, the compression model must operate at any given point along the rate-distortion curve, as specified by a trade-off parameter. Across an independent set of test images, we find that the optimized method generally exhibits better rate-distortion performance than the standard JPEG and JPEG 2000 compression methods. More importantly, we observe a dramatic improvement in visual quality for all images at all bit rates, which is supported by objective quality estimates using MS-SSIM.

Motivation & Objective

• Motivate and address the lossy image compression problem with a learnable nonlinear transform coding framework.
• Introduce end-to-end optimization of analysis/synthesis transforms under a rate–distortion objective.
• Leverage a generalized divisive normalization (GDN) nonlinear gain control to Gaussianize local image statistics.
• Employ a differentiable relaxation of quantization to enable stochastic gradient descent optimization.
• Demonstrate improved rate–distortion performance and substantial perceptual quality gains over JPEG and JPEG 2000.

Proposed method

• Use a three-stage cascade of convolutional filters plus a generalized divisive normalization (GDN) to form the analysis transform.
• Apply a corresponding three-stage synthesis transform with an approximate inverse (IGDN) after uniform scalar quantization in the code space.
• Relax the quantizer with additive uniform noise to enable gradient-based optimization while targeting a rate term based on the entropy of the quantized codes.
• Optimize jointly over analysis/synthesis transforms and the entropy model to minimize a loss L = E[ -log2 p(Ã y) + λ d(z, ẑ) ], approximating rate and distortion via a continuous relaxation.
• Model the code-space marginals p(Ã y) with nonparametric, piecewise-linear densities updated during training.
• Relate the framework to variational autoencoders, highlighting similarities and key differences (discrete compression, end-to-end rate–distortion hull).

Experimental results

Research questions

• RQ1 Can end-to-end optimization of nonlinear transform codes improve rate–distortion performance for natural images?
• RQ2 Does a biologically inspired nonlinear gain control (GDN/IGDN) better Gaussianize image statistics and improve coding efficiency?
• RQ3 What is the impact of optimizing the rate–distortion objective across different λ trade-offs on perceptual quality vs. traditional metrics?
• RQ4 How does the proposed method compare to JPEG and JPEG 2000 in both objective metrics (MS-SSIM, PSNR) and visual quality across bit rates?

Key findings

• The proposed method frequently yields better rate–distortion performance than JPEG and JPEG 2000 on test images.
• At comparable bitrates, the method achieves substantially higher MS-SSIM, indicating superior perceptual quality across images and rates.
• On a representative example, JPEG: 0.121 bit/px with PSNR Luma 24.85 dB and MS-SSIM 0.8079; JPEG 2000: 0.113 bit/px with PSNR Luma 26.61 dB and MS-SSIM 0.8860; Proposed method: 0.113 bit/px with PSNR Luma 27.01 dB and MS-SSIM 0.9039.
• The visual quality advantages include reduced blocking and ringing artifacts compared to linear transform codecs, with smoother contours and preserved edges across bit rates.
• The method demonstrates perceptual improvements across all tested images and bit rates, as evidenced by MS-SSIM-based assessments.
• Training demonstrates that the continuous relaxation provides a good approximation to the discrete rate–distortion objective, enabling effective end-to-end optimization.
• Even though trained with MSE, the approach yields visually superior results, suggesting potential gains if perceptual metrics are used for training.