[Paper Review] Towards Image Understanding from Deep Compression without Decoding

Robert Torfason, Fabian Mentzer+4 more|arXiv (Cornell University)|Mar 16, 2018

Advanced Image Processing TechniquesComputer Science23 references101 citations

TL;DR

The paper shows that image classification and semantic segmentation can be effectively performed directly on compressed representations from learned deep image compression, bypassing RGB decoding and reducing computation, with joint training further improving both compression quality and inference performance.

ABSTRACT

Motivated by recent work on deep neural network (DNN)-based image compression methods showing potential improvements in image quality, savings in storage, and bandwidth reduction, we propose to perform image understanding tasks such as classification and segmentation directly on the compressed representations produced by these compression methods. Since the encoders and decoders in DNN-based compression methods are neural networks with feature-maps as internal representations of the images, we directly integrate these with architectures for image understanding. This bypasses decoding of the compressed representation into RGB space and reduces computational cost. Our study shows that accuracies comparable to networks that operate on compressed RGB images can be achieved while reducing the computational complexity up to $2 imes$. Furthermore, we show that synergies are obtained by jointly training compression networks with classification networks on the compressed representations, improving image quality, classification accuracy, and segmentation performance. We find that inference from compressed representations is particularly advantageous compared to inference from compressed RGB images for aggressive compression rates.

Motivation & Objective

• Understand if compressed representations from learned image compression can support image understanding tasks without decoding to RGB.
• Quantify classification and segmentation performance from compressed representations vs. decoded RGB inputs across multiple bitrate points.
• Explore if joint training of compression and inference networks yields synergistic gains in both reconstruction quality and task accuracy.

Proposed method

• Use a learned convolutional autoencoder for image compression (Theis et al., 2017) producing a compressed feature map of size w/8 × h/8 × C.
• Adapt ResNet variants to operate on compressed representations (cResNet) by discarding early RGB-specific blocks to match 28×28 feature maps.
• Evaluate classification on ImageNet (ILSVRC2012) with 3 operating points: 0.0983, 0.330, 0.635 bpp, comparing compressed-representation inputs vs. reconstructed RGB inputs.
• Evaluate semantic segmentation on PASCAL VOC 2012 with DeepLab-style architecture using dilated convolutions and ASPP, comparing inputs from compressed representations vs. decoded RGB.
• Jointly train compression and inference networks by a combined loss balancing rate–distortion with cross-entropy for classification (Eq. 2).
• Assess computational cost (FLOPs) and performance gains across operating points and architectures.

Experimental results

Research questions

• RQ1 Can inference (classification/segmentation) be performed directly from compressed representations without decoding to RGB?
• RQ2 How does performance from compressed representations compare to decoding in terms of accuracy and computational cost at various bitrate points?
• RQ3 Does joint training of compression and inference networks yield synergistic improvements for both compression quality (SSIM/MS-SSIM) and task accuracy?

Key findings

• Classification from compressed representations achieves near-parity with decoding-based classifiers at similar FLOPs, with 1.5–2× fewer operations required for decoding+inference.
• For ImageNet at 0.635 bpp, top-5 accuracy from compressed representations with cResNet-51 is 87.85%, only 0.5–0.3 percentage points behind RGB-based ResNet-50/ResNet-71 baselines depending on point.
• Segmentation from compressed representations matches or surpasses decoding-based methods at moderate to aggressive compression; at 0.0983 bpp, mean IoU improves by up to 1.65 percentage points over decoded RGB inputs.
• Joint training of compression and classification (and segmentation) yields small but notable gains in perceptual quality (SSIM/MS-SSIM) and improves classification/segmentation accuracy beyond fixed-point baselines (e.g., +2% classification, +1.7% segmentation at 0.0983 bpp, relative to compression-only training).
• Compression + inference networks require minor architectural/training changes and offer substantial runtime/memory benefits when decoding is unnecessary.