[Paper Review] Evaluating Text-to-Visual Generation with Image-to-Text Generation

Zhiqiu Lin, Deepak Pathak+6 more|arXiv (Cornell University)|Apr 1, 2024

Digital Humanities and ScholarshipArts and Humanities3 citations

TL;DR

This paper introduces VQAScore, a novel metric for evaluating text-to-visual generation that uses a visual question answering (VQA) model to compute the probability of a 'Yes' answer to the question 'Does this figure show {text}?'. Despite its simplicity, VQAScore—especially when implemented with a custom bidirectional encoder (CLIP-FlanT5)—achieves state-of-the-art performance across multiple benchmarks, outperforming both CLIPScore and proprietary models like GPT-4V, while also generalizing to video and 3D generation.

ABSTRACT

Despite significant progress in generative AI, comprehensive evaluation remains challenging because of the lack of effective metrics and standardized benchmarks. For instance, the widely-used CLIPScore measures the alignment between a (generated) image and text prompt, but it fails to produce reliable scores for complex prompts involving compositions of objects, attributes, and relations. One reason is that text encoders of CLIP can notoriously act as a "bag of words", conflating prompts such as "the horse is eating the grass" with "the grass is eating the horse". To address this, we introduce the VQAScore, which uses a visual-question-answering (VQA) model to produce an alignment score by computing the probability of a "Yes" answer to a simple "Does this figure show '{text}'?" question. Though simpler than prior art, VQAScore computed with off-the-shelf models produces state-of-the-art results across many (8) image-text alignment benchmarks. We also compute VQAScore with an in-house model that follows best practices in the literature. For example, we use a bidirectional image-question encoder that allows image embeddings to depend on the question being asked (and vice versa). Our in-house model, CLIP-FlanT5, outperforms even the strongest baselines that make use of the proprietary GPT-4V. Interestingly, although we train with only images, VQAScore can also align text with video and 3D models. VQAScore allows researchers to benchmark text-to-visual generation using complex texts that capture the compositional structure of real-world prompts. We introduce GenAI-Bench, a more challenging benchmark with 1,600 compositional text prompts that require parsing scenes, objects, attributes, relationships, and high-order reasoning like comparison and logic. GenAI-Bench also offers over 15,000 human ratings for leading image and video generation models such as Stable Diffusion, DALL-E 3, and Gen2.

Motivation & Objective

• To address the lack of robust, reliable metrics for evaluating text-to-visual generation, especially for complex, compositional prompts.
• To overcome the limitations of existing metrics like CLIPScore, which fail to capture compositional structure due to their 'bag of words' behavior.
• To develop a scalable, reference-free evaluation method that aligns better with human judgment on complex visual reasoning tasks.
• To create a new benchmark, GenAI-Bench, with 1,600 compositional prompts and 15,000 human ratings for leading image and video models.
• To demonstrate that even off-the-shelf VQA models can outperform more complex, proprietary baselines when properly designed and applied.

Proposed method

• Propose VQAScore as a reference-free metric that computes the probability of a 'Yes' answer to the question 'Does this figure show {text}? Please answer yes or no.'
• Use a visual-question-answering (VQA) model to encode both image and question tokens, with a bidirectional encoder allowing mutual attention between image and text.
• Implement VQAScore using off-the-shelf models (e.g., InstructBLIP, LLaVA-1.5) and a custom model, CLIP-FlanT5, which combines CLIP image features with FlanT5 for better cross-attention.
• Apply VQAScore to evaluate text-to-image, text-to-video, and text-to-3D generation by averaging scores across multiple frames or views.
• Design and release GenAI-Bench, a benchmark with 1,600 compositional prompts requiring parsing of objects, attributes, relations, and logical reasoning.
• Conduct ablation studies on frame sampling (for video) and view sampling (for 3D) to show that minimal sampling achieves near-optimal performance.

Experimental results

Research questions

• RQ1 Can a simple VQA-based metric outperform established metrics like CLIPScore in evaluating compositional text-to-visual generation?
• RQ2 To what extent can off-the-shelf VQA models, when properly architected, match or exceed performance of proprietary models like GPT-4V?
• RQ3 How well does VQAScore generalize across different modalities, including video and 3D generation?
• RQ4 Does the use of a bidirectional image-question encoder significantly improve alignment scoring compared to autoregressive encoders?
• RQ5 Can VQAScore achieve high agreement with human judgments on complex, compositional prompts that require reasoning beyond simple object recognition?

Key findings

• VQAScore computed with the CLIP-FlanT5 model achieves a new state-of-the-art pairwise accuracy of 83.1 on the text-to-image benchmark, surpassing even GPT-4V-based methods.
• Using only a single 3x3 grid view of a 3D asset, VQAScore achieves 68.4 pairwise accuracy on StanfordT23D, nearly matching the 68.6 score from 120 views.
• Sampling just four frames from a 3-second video achieves 66.5 pairwise accuracy with CLIP-FlanT5, close to the 66.5 score from all 36 frames.
• VQAScore with off-the-shelf LLaVA-1.5 achieves 82.4 pairwise accuracy on text-to-image benchmarks, outperforming CLIPScore (77.8) and other reference-free metrics.
• The custom CLIP-FlanT5 model achieves 65.4 pairwise accuracy on the text-to-video benchmark EvalCrafter, outperforming InstructBLIP (65.4) and LLaVA-1.5 (63.2).
• Human-annotated ratings on GenAI-Bench show strong correlation with VQAScore, validating its alignment with human judgment on complex, compositional prompts.