[Paper Review] The Rate-Distortion-Accuracy Tradeoff: JPEG Case Study

Xiyang Luo, Hossein Talebi+3 more|arXiv (Cornell University)|Aug 3, 2020

Advanced Data Compression Techniques36 references4 citations

TL;DR

This paper proposes a differentiable, gradient-based optimization framework to jointly tune JPEG quantization tables for improved rate-distortion and rate-accuracy tradeoffs. By modeling JPEG encoding/decoding and bit-rate estimation as differentiable operations, the method enables end-to-end backpropagation to optimize tables for either minimal distortion at a target bit-rate or maximal classification accuracy, achieving up to 17% smaller file sizes at equal quality and improved recognition performance with minimal bit-rate increase.

ABSTRACT

Handling digital images is almost always accompanied by a lossy compression in order to facilitate efficient transmission and storage. This introduces an unavoidable tension between the allocated bit-budget (rate) and the faithfulness of the resulting image to the original one (distortion). An additional complicating consideration is the effect of the compression on recognition performance by given classifiers (accuracy). This work aims to explore this rate-distortion-accuracy tradeoff. As a case study, we focus on the design of the quantization tables in the JPEG compression standard. We offer a novel optimal tuning of these tables via continuous optimization, leveraging a differential implementation of both the JPEG encoder-decoder and an entropy estimator. This enables us to offer a unified framework that considers the interplay between rate, distortion and classification accuracy. In all these fronts, we report a substantial boost in performance by a simple and easily implemented modification of these tables.

Motivation & Objective

• To address the interplay between rate, distortion, and classification accuracy in JPEG compression.
• To improve JPEG performance by optimizing quantization tables beyond default values, which are suboptimal for modern accuracy-driven applications.
• To develop a continuous, differentiable optimization framework that enables gradient-based tuning of quantization tables.
• To evaluate both universal (fixed tables for all images) and per-image (adaptive tables per image) optimization modes.
• To demonstrate that optimized tables can simultaneously improve compression efficiency and recognition accuracy.

Proposed method

• Formalizing the JPEG compression pipeline—including DCT, quantization, entropy coding, and bit-rate estimation—as differentiable functions to enable gradient computation.
• Using mini-batch stochastic gradient descent with backpropagation to optimize quantization tables for a combined loss function balancing rate, distortion, and accuracy.
• Integrating a pre-trained classifier (e.g., ResNet, MobileNet) into the loss function to directly optimize for classification accuracy post-compression.
• Implementing a differentiable entropy estimator to approximate the bit-rate of compressed images without relying on actual encoding.
• Applying the optimization in two modes: universal (one table set for all images) and per-image (individual tables per image).
• Rounding and clipping optimized floating-point tables to integer values (1–255) before final evaluation in the actual JPEG encoder.

Experimental results

Research questions

• RQ1 Can JPEG quantization tables be optimized beyond default values to improve rate-distortion performance?
• RQ2 Can the same optimization framework simultaneously improve rate-accuracy performance for image classification tasks?
• RQ3 How does the performance of optimized tables compare to default tables in terms of file size, PSNR, and classification accuracy?
• RQ4 Does per-image optimization of quantization tables yield better performance than universal table optimization?
• RQ5 How sensitive is the optimization to hyperparameters such as the rate weighting in the loss function?

Key findings

• The proposed method achieved a 17% reduction in file size (from 3.06 bpp to 2.30 bpp) while maintaining the same PSNR (30.24 dB), demonstrating significant rate-distortion improvement.
• For the same image, the optimized table reduced file size by 1.5% while correcting misclassification from 'Bee Eater' to 'Dragonfly' at a lower bit-rate (1.680 bpp vs. 1.706 bpp).
• The optimized quantization tables for rate-distortion performance reduced chroma quantization more than the default tables, reflecting a shift from HVS-based design to distortion-minimization.
• The rate-accuracy performance improved across all bit-rate levels for multiple networks (MobileNetV3, ResNet50, ResNet101), even when the tables were optimized using a different network.
• The optimization was robust to hyperparameter choices, with minimal performance degradation across a wide range of rate-weighting values (c_r from 0.1 to 10.0).
• Per-image optimization outperformed universal optimization, serving as an upper bound for achievable performance in rate-accuracy tradeoffs.