[论文解读] Prompt-ICM: A Unified Framework towards Image Coding for Machines with Task-driven Prompts

Ruoyu Feng, Jinming Liu+4 more|arXiv (Cornell University)|May 4, 2023

Advanced Image Processing Techniques被引用 7

一句话总结

Prompt-ICM 提出了一种面向机器视觉的统一图像编码框架，通过任务驱动的提示（prompt）来优化压缩与特征适配。通过引入用于内容感知比特分配的压缩提示（compression prompts）以及用于下游特征微调的任务自适应提示（task-adaptive prompts），该方法在多种视觉任务中实现了最先进的性能，且额外参数极少。

ABSTRACT

Image coding for machines (ICM) aims to compress images to support downstream AI analysis instead of human perception. For ICM, developing a unified codec to reduce information redundancy while empowering the compressed features to support various vision tasks is very important, which inevitably faces two core challenges: 1) How should the compression strategy be adjusted based on the downstream tasks? 2) How to well adapt the compressed features to different downstream tasks? Inspired by recent advances in transferring large-scale pre-trained models to downstream tasks via prompting, in this work, we explore a new ICM framework, termed Prompt-ICM. To address both challenges by carefully learning task-driven prompts to coordinate well the compression process and downstream analysis. Specifically, our method is composed of two core designs: a) compression prompts, which are implemented as importance maps predicted by an information selector, and used to achieve different content-weighted bit allocations during compression according to different downstream tasks; b) task-adaptive prompts, which are instantiated as a few learnable parameters specifically for tuning compressed features for the specific intelligent task. Extensive experiments demonstrate that with a single feature codec and a few extra parameters, our proposed framework could efficiently support different kinds of intelligent tasks with much higher coding efficiency.

研究动机与目标

• 解决为多样化下游AI任务优化图像压缩而非人类视觉感知的挑战。
• 克服专用编码器和通用特征编码器的局限性，使单一编码器能够支持多种任务。
• 通过减少对任务专用微调或独立编码器的需求，降低计算与存储成本。
• 通过参数高效的微调机制，实现向下游任务的高效迁移。

提出的方法

• 引入压缩提示作为由轻量级信息选择器预测的任务条件化重要性图，以在压缩过程中指导内容加权的比特分配。
• 实现一种空间可变率的特征压缩模型，利用压缩提示根据下游任务将更多比特分配给语义重要区域。
• 设计任务自适应提示作为一组可学习的小型参数，用于微调压缩后的特征以适配特定下游任务。
• 通过联合率失真损失与任务特定损失的端到端训练，实现压缩与下游分析之间的对齐。
• 利用预训练主干网络进行特征提取，并通过信息选择器根据任务语义动态调整压缩策略。
• 采用统一编码器并仅引入极少额外参数，实现跨多种视觉任务的高效部署。

Figure 1: Different pipelines of image coding for machines (ICM): (a). Using the compressed images to support downstream tasks; (b). One-to-one feature-based ICM pipeline; (c). General features based ICM pipelines which ignores the explicit interaction between the compression task and downstream tas

实验结果

研究问题

• RQ1 如何动态地根据不同的下游视觉任务调整图像压缩，以提升编码效率？
• RQ2 任务驱动提示能否在无需专用编码器的情况下，有效协调压缩与下游分析？
• RQ3 一组少量可学习参数（任务自适应提示）在多大程度上可实现与全量微调相当的性能？
• RQ4 压缩提示如何反映任务特定的重要性？其语义对齐性是否可通过可视化验证？

主要发现

• Prompt-ICM 在图像分类、目标检测和语义分割任务中均实现了最先进的性能，且仅使用单一统一编码器。
• 该方法在率失真性能上显著优于 Omni-ICM 和基于预处理的基线方法，同时使用的可学习参数更少。
• 任务自适应提示微调的性能优于全量微调，仅使用 0.87M 参数，而全量微调需 87.61M 参数。
• 压缩提示在视觉上与任务相关区域对齐——根据任务类型聚焦于物体边界或判别性特征。
• 消融实验表明，压缩提示显著提升了率失真性能，尤其在使用多级语义特征时更为明显。
• 该框架在多种任务中保持高性能，且参数开销极小，展现出强大的泛化能力与效率。

Figure 2: The framework of our proposed Prompt-ICM (taking Swin-T as an example). Downstream transferring is performed via task-driven prompt tuning. By tuning the lightweight information selector and task-adaptive prompts, Prompt-ICM could efficiently support various downstream tasks, e.g., classif