[Paper Review] AGIQA-3K: An Open Database for AI-Generated Image Quality Assessment

Chunyi Li, Zicheng Zhang+6 more|arXiv (Cornell University)|Jun 7, 2023

Image Retrieval and Classification Techniques4 citations

TL;DR

This paper introduces AGIQA-3K, the largest open database for AI-generated image (AGI) quality assessment, comprising 2,982 images from six diverse text-to-image models. It establishes fine-grained subjective ratings for perceptual quality and text-to-image alignment, and proposes StairReward—a novel metric that significantly improves alignment assessment performance over existing methods, achieving a PLCC of 0.8713 on the most challenging subset.

ABSTRACT

With the rapid advancements of the text-to-image generative model, AI-generated images (AGIs) have been widely applied to entertainment, education, social media, etc. However, considering the large quality variance among different AGIs, there is an urgent need for quality models that are consistent with human subjective ratings. To address this issue, we extensively consider various popular AGI models, generated AGI through different prompts and model parameters, and collected subjective scores at the perceptual quality and text-to-image alignment, thus building the most comprehensive AGI subjective quality database AGIQA-3K so far. Furthermore, we conduct a benchmark experiment on this database to evaluate the consistency between the current Image Quality Assessment (IQA) model and human perception, while proposing StairReward that significantly improves the assessment performance of subjective text-to-image alignment. We believe that the fine-grained subjective scores in AGIQA-3K will inspire subsequent AGI quality models to fit human subjective perception mechanisms at both perception and alignment levels and to optimize the generation result of future AGI models. The database is released on https://github.com/lcysyzxdxc/AGIQA-3k-Database.

Motivation & Objective

• To address the lack of comprehensive, fine-grained subjective quality databases for AI-generated images (AGIs) across diverse models and prompts.
• To establish a standardized, large-scale subjective evaluation framework for perceptual quality and text-to-image alignment in AGIs.
• To benchmark existing Image Quality Assessment (IQA) models against human perception and identify performance gaps.
• To propose and validate StairReward, a new metric that enhances the objective assessment of text-to-image alignment in AGIs.
• To provide a foundation for future AGI quality models that better align with human perceptual mechanisms.

Proposed method

• The authors collected 2,982 AGIs using six different text-to-image models, including GAN-based, autoregressive, and diffusion-based architectures, across varied prompts and model parameters.
• A standardized, laboratory-based subjective experiment was conducted to collect Mean Opinion Scores (MOS) for both perceptual quality and text-to-image alignment.
• The database includes fine-grained annotations across multiple dimensions, covering diverse styles and prompt complexities to ensure broad representativeness.
• StairReward was proposed as a new alignment metric using prompt segmentation and image patch-level feature analysis to better capture semantic alignment.
• Benchmark experiments were conducted on AGIQA-3K to evaluate the correlation (SRoCC, KRoCC, PLCC) between objective metrics and human subjective scores.
• An ablation study was performed to validate the contribution of each component in StairReward, confirming the importance of both word-level segmentation and image patch processing.

Experimental results

Research questions

• RQ1 How do existing IQA models perform in assessing perceptual quality and text-to-image alignment of AGIs compared to human subjective ratings?
• RQ2 To what extent can current metrics distinguish between AGIs of similar quality, especially within the same generation model?
• RQ3 Can a new metric be designed to significantly improve the objective assessment of text-to-image alignment in AGIs?
• RQ4 What components are most critical for improving alignment assessment performance, and how do they contribute individually?
• RQ5 How does the performance of alignment metrics vary across different prompt styles and image content types?

Key findings

• StairReward achieved a PLCC of 0.8713 on the 'No Style' subset, significantly outperforming existing metrics like ImageReward (PLCC: 0.7743) and CLIP (PLCC: 0.6867).
• The proposed StairReward outperformed all baseline metrics in all four prompt-length subsets, with the highest SRoCC (0.7682) and PLCC (0.8468) on the 'Abstract & Sci-fi Style' subset.
• Existing perception-based IQA models showed strong correlation with human perception (PLCC > 0.7), but alignment models lagged significantly, indicating a critical need for improvement.
• The ablation study confirmed that both prompt segmentation and image patch-level processing are essential, as removing either component degraded performance across all metrics.
• The database reveals that even within the same model, image quality varies widely based on prompts and parameters, underscoring the need for robust, fine-grained assessment tools.
• AGIQA-3K provides the most comprehensive, multi-model, multi-dimensional AGI quality database to date, covering GAN, autoregressive, and diffusion-based models with fine-grained subjective ratings.