[Paper Review] Large Scale Qualitative Evaluation of Generative Image Model Outputs

Yannick Assogba, Adam Pearce+1 more|arXiv (Cornell University)|Jan 11, 2023

Aesthetic Perception and Analysis4 citations

TL;DR

This paper introduces Ravel, a visual analytics system that enables large-scale qualitative evaluation of generative image model outputs across hundreds of thousands of images. By leveraging clustering and semantic embedding spaces, Ravel supports interactive exploration to detect issues like mode collapse, missing data distribution regions, and quality/diversity trade-offs—offering insights beyond standard metrics like FID.

ABSTRACT

Evaluating generative image models remains a difficult problem. This is due to the high dimensionality of the outputs, the challenging task of representing but not replicating training data, and the lack of metrics that fully correspond to human perception and capture all the properties we want these models to exhibit. Therefore, qualitative evaluation of model outputs is an important part of model development and research publication practice. Quantitative evaluation is currently under-served by existing tools, which do not easily facilitate structured exploration of a large number of examples across the latent space of the model. To address this issue, we present Ravel, a visual analytics system that enables qualitative evaluation of model outputs on the order of hundreds of thousands of images. Ravel allows users to discover phenomena such as mode collapse, and find areas of training data that the model has failed to capture. It allows users to evaluate both quality and diversity of generated images in comparison to real images or to the output of another model that serves as a baseline. Our paper describes three case studies demonstrating the key insights made possible with Ravel, supported by a domain expert user study.

Motivation & Objective

• To address the lack of scalable tools for qualitative evaluation of generative image models, which are critical for detecting issues like mode collapse and data distribution gaps.
• To support researchers in exploring model outputs at scale—up to 120k images—beyond the typical 10s to 100s of images examined in practice.
• To provide a system-agnostic interface that enables comparison of image quality, diversity, and distributional fidelity against real data or baseline models.
• To facilitate hypothesis generation about model behavior through visual comparison in semantic embedding spaces.
• To overcome limitations of single-number metrics like FID by enabling fine-grained, human-guided inspection of model outputs at scale.

Proposed method

• Ravel uses a two-stage approach: first, it clusters generated images using learned embeddings to group visually and semantically similar samples.
• Second, it visualizes clusters with aggregated metrics (e.g., FID, precision, recall) to guide users in identifying problematic regions.
• The system supports interactive exploration through a user interface that enables side-by-side comparison of real and generated images using semantic embedding spaces.
• It allows users to navigate clusters, inspect individual images at high resolution, and explore outliers within clusters to detect artifacts or distributional failures.
• The interface supports both global and local views: users can switch between class-conditional views and similarity-based clustering to explore model behavior from different perspectives.
• Ravel is designed to be model-agnostic, working with outputs from any generative model architecture without requiring access to model internals.

Figure 1: The Ravel interface primarily consists of: A) Dataset & view options. B) Summary charts & linked cluster plots. C) Side by side image grids for visual comparison of clusters. This view shows a cluster comparing real images on the left to generated images on the right.

Experimental results

Research questions

• RQ1 How can researchers effectively explore and evaluate the quality and diversity of generative image model outputs at scale, beyond small-sample qualitative inspection?
• RQ2 What visual analytics techniques enable users to detect mode collapse and missing regions of the training data distribution in large-scale model outputs?
• RQ3 To what extent can visual comparison in semantic embedding spaces support hypothesis generation about model behavior and failure modes?
• RQ4 How do human experts use large-scale visual exploration to identify issues that single-number metrics like FID fail to capture?
• RQ5 What limitations or usability challenges arise when using clustering and embedding-based navigation for qualitative evaluation of generative models?

Key findings

• Experts using Ravel identified previously undetected gaps in model coverage, such as missing face paint and specific headgear in a StyleGAN2-based model, which were not apparent from FID scores.
• The system enabled detection of mode collapse in BigGAN-deep, where generated samples were limited to a narrow subset of the training distribution, a failure not captured by standard metrics.
• Users consistently reported that visual inspection in Ravel revealed artifacts and distributional issues—such as texture bias or memorization—more effectively than quantitative metrics alone.
• Participants found that clustering by similarity, rather than class label, was more effective for identifying outliers and rare failure modes, suggesting that similarity-based grouping enhances diagnostic insight.
• Despite its strengths, users struggled to interpret cluster semantics without additional summarization, indicating a need for better cluster description or hierarchical clustering support in future versions.
• A key limitation identified was the inability to detect memorization in real time; users suggested integrating nearest-neighbor search to enhance this capability.

Figure 2: Beeswarm plot showing distribution of cluster precision scores. Each dot is a cluster which the currently selected dot shown in orange. A description of the metric can be accessed by clicking on the ? icon.