[Paper Review] Semantic-assisted image compression

Qizheng Sun, Caili Guo+3 more|arXiv (Cornell University)|Jan 29, 2022

Advanced Data Compression Techniques4 citations

TL;DR

This paper proposes Semantic-Assisted Image Compression (SAIC), a novel deep learning-based image compression framework that optimizes for semantic-level consistency in downstream AI tasks rather than pixel-level fidelity. By leveraging a gradient-based semantic weights mechanism (GSW) and a semantic mutual information (SI) metric, SAIC achieves superior performance in classification and object detection tasks at equivalent compression ratios, outperforming both traditional and perceptual compression methods.

ABSTRACT

Conventional image compression methods typically aim at pixel-level consistency while ignoring the performance of downstream AI tasks.To solve this problem, this paper proposes a Semantic-Assisted Image Compression method (SAIC), which can maintain semantic-level consistency to enable high performance of downstream AI tasks.To this end, we train the compression network using semantic-level loss function. In particular, semantic-level loss is measured using gradient-based semantic weights mechanism (GSW). GSW directly consider downstream AI tasks' perceptual results. Then, this paper proposes a semantic-level distortion evaluation metric to quantify the amount of semantic information retained during the compression process. Experimental results show that the proposed SAIC method can retain more semantic-level information and achieve better performance of downstream AI tasks compared to the traditional deep learning-based method and the advanced perceptual method at the same compression ratio.

Motivation & Objective

• To address the limitation of conventional image compression methods that prioritize pixel-level consistency over downstream AI task performance.
• To maintain semantic-level consistency during image compression to improve performance in tasks like image classification and object detection.
• To develop a method that directly incorporates perceptual results from downstream AI models into the compression training process.
• To introduce a new metric, semantic mutual information (SI), to quantify semantic-level distortion during compression.

Proposed method

• The SAIC framework uses a compression network trained with a semantic-level loss function derived from perceptual outputs of downstream AI tasks.
• A gradient-based semantic weights mechanism (GSW) dynamically assigns importance to feature maps based on their contribution to downstream task performance.
• The GSW mechanism computes attention weights using gradients from the final layer of the downstream model, focusing on features most relevant to semantic understanding.
• A new semantic mutual information (SI) metric quantifies the amount of semantic information retained after compression, serving as a distortion measure.
• The method integrates the GSW and SI into the end-to-end training of the compression autoencoder, enabling joint optimization of compression and task performance.
• The approach generalizes across tasks, as demonstrated on classification and object detection benchmarks with varying compression ratios.

Experimental results

Research questions

• RQ1 Can image compression be optimized to preserve semantic-level information critical for downstream AI tasks?
• RQ2 How can perceptual outputs from downstream models be effectively used to guide the compression process?
• RQ3 What is the impact of semantic-level optimization on task performance compared to pixel-level or feature-level optimization?
• RQ4 Can a new distortion metric based on semantic mutual information (SI) accurately reflect the quality of compressed images for AI tasks?

Key findings

• At 0.125 bpp, SAIC achieved 69.69% classification accuracy on the STL-10 dataset, outperforming TDIC (63.26%) and APIC (68.55%) by 6.43% and 1.14% respectively.
• On the same dataset, SAIC achieved a semantic mutual information (SI) score of 68.10 at 0.125 bpp, significantly higher than TDIC (53.87) and APIC (64.60).
• For object detection on Pascal VOC at 0.5 bpp, SAIC maintained a mean average precision (mAP) of 72.0%, comparable to APIC (72.1%) and substantially better than TDIC (71.3%).
• The performance improvement of SAIC becomes more pronounced at lower bitrates, with the highest relative gains observed at 0.125 bpp, indicating strong robustness under high compression.
• The optimal temperature hyper-parameter τ for GSW was found to be around 2900–3100, balancing weight distribution to avoid over-concentration or dispersion.
• The proposed SI metric effectively quantifies semantic-level distortion and correlates with downstream task performance, validating its use as a meaningful evaluation metric.