[Paper Review] Approximating Human Judgment of Generated Image Quality

Y. Alex Kolchinski, Sharon Zhou+3 more|arXiv (Cornell University)|Nov 30, 2019

Generative Adversarial Networks and Image Synthesis28 references4 citations

TL;DR

This paper proposes a novel automated method to predict human judgment of image realism by regressing deep feature embeddings from an Inception-V3 network to human-labeled quality scores. It achieves 66.4% accuracy on held-out test sets—matching human inter-rater agreement—demonstrating strong alignment with perceptual quality across diverse generative models.

ABSTRACT

Generative models have made immense progress in recent years, particularly in their ability to generate high quality images. However, that quality has been difficult to evaluate rigorously, with evaluation dominated by heuristic approaches that do not correlate well with human judgment, such as the Inception Score and Fréchet Inception Distance. Real human labels have also been used in evaluation, but are inefficient and expensive to collect for each image. Here, we present a novel method to automatically evaluate images based on their quality as perceived by humans. By not only generating image embeddings from Inception network activations and comparing them to the activations for real images, of which other methods perform a variant, but also regressing the activation statistics to match gold standard human labels, we demonstrate 66% accuracy in predicting human scores of image realism, matching the human inter-rater agreement rate. Our approach also generalizes across generative models, suggesting the potential for capturing a model-agnostic measure of image quality. We open source our dataset of human labels for the advancement of research and techniques in this area.

Motivation & Objective

• Address the lack of efficient, sample-level metrics that correlate with human perception of image realism in generative models.
• Overcome limitations of existing automated metrics like Inception Score and FID, which fail to correlate well with human judgment at the sample level.
• Develop a model-agnostic evaluation method that generalizes across different generative models and datasets.
• Bridge the gap between automated metrics and human perception by training on both deep feature embeddings and human-annotated realism scores.
• Enable scalable, reliable evaluation of image quality without relying on costly human labeling for every image.

Proposed method

• Collect human realism judgments using the HYPE framework, which presents images in binary real/fake classification tasks to estimate perceptual realism.
• Extract multi-layer feature embeddings from an ImageNet-pretrained Inception-V3 network across seven layers (e.g., Mixed_5d, Mixed_6e, FC).
• For each test image, compute spatially localized nearest-neighbor distances between its activation maps and those of real images in the training set.
• Aggregate these distances across spatial locations to form a single real-valued embedding per layer, creating a multi-dimensional representation per image.
• Train a regression model to map these multi-layer embeddings to human-labeled realism scores, using both feature similarity and human judgment as supervision.
• Use the final model to predict sample-level realism scores that reflect human perception, with performance evaluated via classification accuracy against human labels.

Experimental results

Research questions

• RQ1 Can a deep learning model trained on both deep feature embeddings and human-labeled realism scores accurately predict human judgments of image quality?
• RQ2 To what extent does the proposed method generalize across different generative models (e.g., ProGAN vs. StyleGAN) and unseen data distributions?
• RQ3 How well does the model’s performance correlate with human inter-rater agreement on image realism?
• RQ4 Can the method achieve sample-level evaluation that correlates strongly with human perception, unlike population-level metrics such as FID or IS?
• RQ5 Does combining feature-based similarity with human-annotated scores improve the predictive power of automated image quality metrics?

Key findings

• The proposed method achieves 66.4% accuracy in predicting human-labeled image realism on held-out test sets from the same generative model (StyleGAN), matching the upper end of human inter-rater agreement (65.5%).
• On out-of-distribution samples from a different generative model (ProGAN), the model achieves 58.8% accuracy, closely aligning with human inter-rater agreement of 58.9%.
• The model generalizes across generative models, indicating potential for a model-agnostic measure of image quality.
• Performance is significantly improved by combining multi-layer feature embeddings with human-annotated scores, outperforming methods relying solely on feature statistics.
• The method provides sample-level realism scores, unlike FID or IS, which only evaluate model-level performance on large distributions.
• The authors release their human-annotated dataset to support future research in human-aligned image quality evaluation.