[Paper Review] Improving Automatic VQA Evaluation Using Large Language Models

Oscar Mañas, Benno Krojer+1 more|arXiv (Cornell University)|Oct 4, 2023

Multimodal Machine Learning ApplicationsComputer Science3 citations

TL;DR

This paper proposes LAVE, a novel automatic VQA evaluation metric that uses instruction-tuned large language models (LLMs) to rate answer correctness by comparing candidate answers against reference answers via in-context learning. LAVE significantly outperforms VQA Accuracy and soft metrics like BERTScore in correlation with human judgment, especially in out-of-distribution and paraphrase-heavy settings, while providing interpretable rationales for its scores.

ABSTRACT

8 years after the visual question answering (VQA) task was proposed, accuracy remains the primary metric for automatic evaluation. VQA Accuracy has been effective so far in the IID evaluation setting. However, our community is undergoing a shift towards open-ended generative models and OOD evaluation. In this new paradigm, the existing VQA Accuracy metric is overly stringent and underestimates the performance of VQA systems. Thus, there is a need to develop more robust automatic VQA metrics that serve as a proxy for human judgment. In this work, we propose to leverage the in-context learning capabilities of instruction-tuned large language models (LLMs) to build a better VQA metric. We formulate VQA evaluation as an answer-rating task where the LLM is instructed to score the accuracy of a candidate answer given a set of reference answers. We demonstrate the proposed metric better correlates with human judgment compared to existing metrics across several VQA models and benchmarks. We hope wide adoption of our metric will contribute to better estimating the research progress on the VQA task. We plan to release the evaluation code and collected human judgments.

Motivation & Objective

• To address the limitations of VQA Accuracy in out-of-distribution (OOD) evaluation settings where models generate correct but non-matching answers.
• To develop a more robust, human-judgment-aligned automatic metric that accounts for semantic equivalence beyond exact string matching.
• To leverage the reasoning and language understanding capabilities of instruction-tuned LLMs for improved answer evaluation in vision-language tasks.
• To systematically evaluate and compare LAVE against existing metrics, including VQA Accuracy and soft metrics like BERTScore.
• To provide an interpretable, scalable, and reliable alternative to human evaluation for benchmarking VQA models.

Proposed method

• Formulate VQA evaluation as an answer-rating task where an instruction-tuned LLM scores the correctness of a candidate answer given a question and a set of reference answers.
• Use in-context learning to prompt the LLM with few-shot demonstrations of correct/incorrect answer pairs to guide its judgment.
• Design prompts that include the question, candidate answer, and reference answers, instructing the LLM to rate correctness on a 0–1 scale.
• Employ multiple LLMs (e.g., Flan-T5, GPT-3.5) to evaluate robustness and generalization across model types.
• Collect human judgments on 22,100 test examples across three VQA benchmarks to serve as the gold standard for correlation analysis.
• Use correlation coefficients (e.g., Spearman’s ρ) to quantitatively compare LAVE’s performance against baseline metrics.

Experimental results

Research questions

• RQ1 How does LAVE’s correlation with human judgment compare to that of VQA Accuracy and soft metrics like BERTScore across diverse VQA models and benchmarks?
• RQ2 In what ways does LAVE recover correct answers that are missed by VQA Accuracy due to paraphrasing, format differences, or lexical variation?
• RQ3 How robust is LAVE to variations in answer verbosity, phrasing, and semantic equivalence across different VQA evaluation settings?
• RQ4 What is the impact of different LLM architectures (e.g., Flan-T5 vs. GPT-3.5) on the performance and reliability of the LAVE metric?
• RQ5 How do failure modes of VQA Accuracy manifest, and to what extent does LAVE mitigate them?

Key findings

• LAVE achieves significantly higher correlation with human judgment than VQA Accuracy and soft metrics like BERTScore, especially in OOD and paraphrase-heavy settings.
• LAVE recovers 16.33% of cases where VQA Accuracy incorrectly labeled correct answers as incorrect due to lexical or syntactic differences.
• In 8.25% of cases, LAVE correctly rates semantically correct answers as correct (score 1.0) even when VQA Accuracy yields scores below 0.5, demonstrating improved robustness.
• LAVE outperforms all baselines, including Soft VQA Accuracy and BERTScore, in terms of Spearman’s correlation with human judgments across all evaluated benchmarks.
• Ablation studies confirm that both the instruction design and the use of reference answers are critical components for LAVE’s performance, with the former having a stronger impact.
• LAVE provides interpretable rationales for its scores, enhancing transparency and trust in automated evaluation.