[论文解读] Image Quality Assessment for Perceptual Image Restoration: A New Dataset, Benchmark and Metric

Jinjin Gu, Haoming Cai+4 more|arXiv (Cornell University)|Nov 30, 2020

Advanced Image Processing Techniques参考文献 50被引用 31

一句话总结

这篇论文引入了带有GAN基础IR输出的PIPAL数据集、基于 Elo 的 MOS，以及用于基准和改进感知图像恢复（IQA）方法的空间扭曲差分IQA网络（SWDN），用于评测和提升IQA方法在感知图像恢复中的表现。它显示当前IQA指标在GAN扭曲图像上表现不佳，并提出改进措施。

ABSTRACT

Image quality assessment (IQA) is the key factor for the fast development of image restoration (IR) algorithms. The most recent perceptual IR algorithms based on generative adversarial networks (GANs) have brought in significant improvement on visual performance, but also pose great challenges for quantitative evaluation. Notably, we observe an increasing inconsistency between perceptual quality and the evaluation results. We present two questions: Can existing IQA methods objectively evaluate recent IR algorithms? With the focus on beating current benchmarks, are we getting better IR algorithms? To answer the questions and promote the development of IQA methods, we contribute a large-scale IQA dataset, called Perceptual Image Processing ALgorithms (PIPAL) dataset. Especially, this dataset includes the results of GAN-based IR algorithms, which are missing in previous datasets. We collect more than 1.13 million human judgments to assign subjective scores for PIPAL images using the more reliable Elo system. Based on PIPAL, we present new benchmarks for both IQA and SR methods. Our results indicate that existing IQA methods cannot fairly evaluate GAN-based IR algorithms. While using appropriate evaluation methods is important, IQA methods should also be updated along with the development of IR algorithms. At last, we shed light on how to improve the IQA performance on GAN-based distortion. Inspired by the find that the existing IQA methods have an unsatisfactory performance on the GAN-based distortion partially because of their low tolerance to spatial misalignment, we propose to improve the performance of an IQA network on GAN-based distortion by explicitly considering this misalignment. We propose the Space Warping Difference Network, which includes the novel l_2 pooling layers and Space Warping Difference layers. Experiments demonstrate the effectiveness of the proposed method.

研究动机与目标

• 激发对GAN为基础的感知图像恢复（IR）所带来评估挑战。
• 提出一个包含GAN型失真的大规模IQA数据集（PIPAL）.
• 通过基于Elo的MOS和一个开放评定工具（IQOS）提供可靠的主观分数。
• 对IR任务中的现有IQA方法进行基准测试，以发现差距。
• 通过架构改进提出对GAN失真 的IQA改进。

提出的方法

• 创建PIPAL，包含29k张扭曲图像（包括基于GAN的输出）以及跨250个参考的116种失真类型。
• 使用Elo评定系统收集主观分数，生成具有大量人类判断的MOS（>1.13 million）。
• 开发IQOS，一个结合Swiss和Elo评级的基于网络的打分系统，以实现可扩展的注释。
• 在PIPAL上对一系列FR-IQA与NR-IQA方法（PSNR、SSIM、LPIPS、PieAPP、DISTS、WaDIQaM、NIQE、PI等）相对于MOS进行评估。
• 分析GAN失真特征并指出空间错位是许多IQA方法的一个关键弱点。
• 提出带有l2-池化和Space Warping Difference (SWD)层的Space Warping Difference Network (SWDN)，以提高对错位的鲁棒性。

实验结果

研究问题

• RQ1 现有IQA方法是否能够客观评估基于GAN的感知IR输出？
• RQ2 包括GAN方法在内的现代IR算法，当前的IQA指标是否合适作为基准？
• RQ3 在IQA性能方面，GAN基失真与传统失真有何不同？
• RQ4 通过解决GAN失真中的空间错位的架构改动，是否可以改进IQA性能？

主要发现

• 在PIPAL中，现有IQA方法很难与人类判断在GAN基失真上的相关性。
• PieAPP、LPIPS和WaDIQaM对GAN基失真显示相对更好的对齐，但仍然表现不足。
• GAN基失真导致传统FR-IQA指标（PSNR、SSIM等）相关性大幅下降。
• 空间错位是降低GAN失真IQA性能的一个主要因素；解决它可以提升鲁棒性。
• 采用l2- pooling和SWD层的SWDN在IQA任务的GAN基失真上实现了最先进的性能。