[Paper Review] Classification Accuracy Score for Conditional Generative Models

Suman Ravuri, Oriol Vinyals|arXiv (Cornell University)|May 26, 2019

Generative Adversarial Networks and Image SynthesisComputer Science44 references99 citations

TL;DR

Classification Accuracy Score (CAS) trains a classifier on synthetic data from conditional generative models and tests on real data to measure downstream task performance, revealing weaknesses not captured by IS/FID and showing likelihood-based models can outperform GANs on CAS.

ABSTRACT

Deep generative models (DGMs) of images are now sufficiently mature that they produce nearly photorealistic samples and obtain scores similar to the data distribution on heuristics such as Frechet Inception Distance (FID). These results, especially on large-scale datasets such as ImageNet, suggest that DGMs are learning the data distribution in a perceptually meaningful space and can be used in downstream tasks. To test this latter hypothesis, we use class-conditional generative models from a number of model classes---variational autoencoders, autoregressive models, and generative adversarial networks (GANs)---to infer the class labels of real data. We perform this inference by training an image classifier using only synthetic data and using the classifier to predict labels on real data. The performance on this task, which we call Classification Accuracy Score (CAS), reveals some surprising results not identified by traditional metrics and constitute our contributions. First, when using a state-of-the-art GAN (BigGAN-deep), Top-1 and Top-5 accuracy decrease by 27.9\% and 41.6\%, respectively, compared to the original data; and conditional generative models from other model classes, such as Vector-Quantized Variational Autoencoder-2 (VQ-VAE-2) and Hierarchical Autoregressive Models (HAMs), substantially outperform GANs on this benchmark. Second, CAS automatically surfaces particular classes for which generative models failed to capture the data distribution, and were previously unknown in the literature. Third, we find traditional GAN metrics such as Inception Score (IS) and FID neither predictive of CAS nor useful when evaluating non-GAN models. Furthermore, in order to facilitate better diagnoses of generative models, we open-source the proposed metric.

Motivation & Objective

• Motivate evaluation of generative models by downstream task performance rather than solely perceptual metrics like IS/FID.
• Define and formalize the Classification Accuracy Score (CAS) as a measure of how well synthetic data supports labeled classification compared to real data.
• Compare multiple model classes (GANs, VQ-VAE-2, HAMs) on CAS on large-scale (ImageNet) and smaller-scale (CIFAR-10) datasets.
• Demonstrate that CAS can reveal class-specific deficiencies and that traditional GAN metrics do not predict CAS well.
• Open-source the CAS metric to facilitate broader adoption and diagnostic use.

Proposed method

• Train an image classifier (ResNet-based) on synthetic data produced by conditional generative models.
• Evaluate the classifier on real data to obtain Top-1 and Top-5 accuracy, defining CAS.
• Compare CAS to Inception Score (IS) and Frechet Inception Distance (FID) across model classes.
• Perform per-class analysis to identify which categories are poorly captured by each model.
• Introduce Naive Augmentation Score (NAS) by training classifiers on mixtures of real and synthetic data to study augmentation effects.
• Open-source the CAS computation workflow for reproducibility and broader use.

Experimental results

Research questions

• RQ1 Can CAS reveal downstream-task deficiencies in conditional generative models that IS/FID miss?
• RQ2 Which model classes (GANs vs likelihood-based models like VQ-VAE-2 and HAMs) achieve higher CAS on ImageNet and CIFAR-10?
• RQ3 Are there specific classes where generative models consistently fail to capture the data distribution, as revealed by per-class CAS?
• RQ4 How does CAS relate to traditional metrics (IS, FID) across model families?
• RQ5 Does augmenting real data with model-generated samples (NAS) improve downstream task performance, and under what conditions?

Key findings

• BigGAN-deep on ImageNet shows substantial CAS degradation (Top-1 drop of 27.9% and Top-5 drop of 41.6%) relative to real data.
• Likelihood-based conditional models (VQ-VAE-2, HAM) achieve higher CAS than BigGAN-deep despite poorer IS/FID.
• CAS per-class analysis identifies specific categories where BigGAN-deep and other models fail to capture the data distribution (e.g., balloon, paddlewheel, pencil sharpener, spatula with 0% accuracy in some cases).
• IS and FID do not reliably predict CAS, especially for non-GAN models, highlighting the need for task-aligned evaluation metrics.
• Naive Augmentation Score (NAS) can yield small classification gains (e.g., up to ~0.2% Top-5) when augmenting real data with synthetic samples; however, results vary with truncation and model.