[Paper Review] GIQA: Generated Image Quality Assessment

Shuyang Gu, Jianmin Bao+2 more|arXiv (Cornell University)|Mar 19, 2020

Advanced Image Processing Techniques41 references4 citations

TL;DR

This paper introduces Generated Image Quality Assessment (GIQA), a novel framework for objectively evaluating the quality of individual generated images from GANs. It proposes three methods—MBC-GIQA (learning-based with semi-supervised regression), GMM-GIQA (parametric data-based), and KNN-GIQA (non-parametric data-based)—demonstrating strong consistency with human judgments and enabling applications like separate realism/diversity evaluation and OHEM-based GAN training improvements.

ABSTRACT

Generative adversarial networks (GANs) have achieved impressive results today, but not all generated images are perfect. A number of quantitative criteria have recently emerged for generative model, but none of them are designed for a single generated image. In this paper, we propose a new research topic, Generated Image Quality Assessment (GIQA), which quantitatively evaluates the quality of each generated image. We introduce three GIQA algorithms from two perspectives: learning-based and data-based. We evaluate a number of images generated by various recent GAN models on different datasets and demonstrate that they are consistent with human assessments. Furthermore, GIQA is available to many applications, like separately evaluating the realism and diversity of generative models, and enabling online hard negative mining (OHEM) in the training of GANs to improve the results.

Motivation & Objective

• Address the lack of methods to assess the quality of individual generated images from GANs, which are often inconsistent in realism and artifact quality.
• Overcome limitations of existing metrics like FID and Inception Score, which evaluate entire models rather than individual samples.
• Develop a no-reference, automated quality assessment system tailored specifically for generated images, distinct from traditional NR-IQA methods.
• Enable new applications such as online hard negative mining in GAN training and selective image picking based on quality.
• Provide a benchmark with a new labeled dataset (LGIQA) to evaluate GIQA performance objectively.

Proposed method

• Propose MBC-GIQA, a semi-supervised learning-based method using multiple binary classifiers to regress image quality scores, leveraging training iteration count as pseudo-labels.
• Introduce GMM-GIQA, a parametric data-based method that models the real image distribution using a Gaussian Mixture Model and assigns quality scores based on likelihood.
• Develop KNN-GIQA, a non-parametric data-based method that estimates image quality by computing the distance to its K nearest neighbors in the real image space.
• Use a novel multi-binary-classifier regression strategy to reduce label noise in the semi-supervised setting, improving robustness without manual annotations.
• Apply OHEM (Online Hard Negative Mining) in GAN training by weighting loss higher for low-quality generated images, as identified by GIQA.
• Leverage GIQA as an image picker to select high-quality samples from a generated batch for downstream use.

Experimental results

Research questions

• RQ1 Can a no-reference, automated method accurately assess the quality of individual generated images without human annotations?
• RQ2 How do learning-based and data-based approaches compare in performance and efficiency for GIQA?
• RQ3 Can GIQA be used to disentangle and separately evaluate realism and diversity in generative models?
• RQ4 To what extent can GIQA improve GAN training through OHEM by focusing on low-quality samples?
• RQ5 How sensitive are the proposed GIQA methods to hyperparameters such as patch size, number of components, or K in KNN?

Key findings

• MBC-GIQA achieves 73.1% accuracy on LGIQA-FFHQ with a patch size of 192 and 8 binary classifiers, peaking before degradation at higher numbers.
• GMM-GIQA performs best with 70 Gaussian components (76.4% accuracy), showing optimal performance before degradation at 100 components.
• KNN-GIQA achieves 84.3% accuracy with K=3500 nearest neighbors, with performance stabilizing from K=500 to K=7000.
• Applying OHEM with GIQA-based weighting improves FID and quality scores, with further gains when combined with an image picker strategy.
• The proposed GIQA methods show strong consistency with human evaluations, outperforming NIMA on score distribution alignment.
• The framework enables effective separation of realism and diversity assessment, offering a more nuanced evaluation than FID or Inception Score.