[論文レビュー] A Comprehensive Evaluation of GPT-4V on Knowledge-Intensive Visual Question Answering

Yunxin Li, Longyue Wang+6 more|arXiv (Cornell University)|Nov 13, 2023

Multimodal Machine Learning Applications被引用数 8

ひとこと要約

tldr: The paper benchmarks GPT-4V on knowledge-intensive VQA across commonsense, fine-grained world knowledge, and decision-making rationales, showing SOTA performance but highlighting weaknesses in world-knowledge hallucinations and reliance on visual cues.

ABSTRACT

The emergence of multimodal large models (MLMs) has significantly advanced the field of visual understanding, offering remarkable capabilities in the realm of visual question answering (VQA). Yet, the true challenge lies in the domain of knowledge-intensive VQA tasks, which necessitate not just recognition of visual elements, but also a deep comprehension of the visual information in conjunction with a vast repository of learned knowledge. To uncover such capabilities of MLMs, particularly the newly introduced GPT-4V and Gemini, we provide an in-depth evaluation from three perspectives: 1) Commonsense Knowledge, which assesses how well models can understand visual cues and connect to general knowledge; 2) Fine-grained World Knowledge, which tests the model's skill in reasoning out specific knowledge from images, showcasing their proficiency across various specialized fields; 3) Comprehensive Knowledge with Decision-making Rationales, which examines model's capability to provide logical explanations for its inference, facilitating a deeper analysis from the interpretability perspective. Additionally, we utilize a visual knowledge-enhanced training strategy and multimodal retrieval-augmented generation approach to enhance MLMs, highlighting the future need for advancements in this research direction. Extensive experiments indicate that: a) GPT-4V demonstrates enhanced explanation generation when using composite images as few-shots; b) GPT-4V and other MLMs produce severe hallucinations when dealing with world knowledge; c) Visual knowledge enhanced training and prompting technicals present potential to improve performance. Codes: https://github.com/HITsz-TMG/Cognitive-Visual-Language-Mapper

研究の動機と目的

• Assess GPT-4V and other MLMs on commonsense knowledge VQA using OK-VQA-derived prompts and composite-image prompting.
• Evaluate GPT-4V on fine-grained world knowledge with INFOSEEK-derived samples across multiple domains.
• Examine GPT-4V’s ability to generate decision-making rationales using A-OKVQA rationales.
• Compare prompting strategies (few-shot vs zero-shot) and analyze reasoning ability and interpretability.
• Identify common failure modes and provide insights for improving knowledge-grounded multimodal models.

提案手法

• Construct a knowledge-intensive VQA benchmark by reconfiguring OK-VQA, INFOSEEK, and A-OKVQA datasets into evaluation subsets.
• Use exact matching for short answers and automatic metrics (BLEU, CIDER, METEOR) plus human evaluation for rationales.
• Employ composite in-context prompting for GPT-4V to provide reference samples across commonsense, physical, world, and visual knowledge.
• Evaluate a range of MLMs (open-source and GPT-4V) in zero-shot and few-shot settings.
• Analyze reasoning via decision-making rationales and assess interpretability with human judgments on consistency, sufficiency, and factual correctness.
• Examine prompting method efficacy and the impact of in-context reference examples embedded in a composite image.

実験結果

リサーチクエスチョン

• RQ1 マルチモーダル大規模モデルは多様なカテゴリーにわたる常識知識VQAでどの程度性能を発揮するか？
• RQ2 エンティティ特異情報を要するVQAタスクにおいて、モデルはファイングレインな世界知識をどの程度正確に扱えるか？
• RQ3 GPT-4Vと同輩はVQAの解答に対する信頼できる意思決定根拠を生成できるか？
• RQ4 プロンプティング戦略（ゼロショット対ファウショット）と複合画像プロンプティングは性能と解釈性にどのような影響を与えるか？
• RQ5 知識集約型のVQAにおけるGPT-4Vの主な失敗モードは何か、どのように緩和できるか？

主な発見

• GPT-4Vは常識知識、ファイングレインな世界知識、根拠生成タスクで最先端の性能を達成。
• GPT-4Vは複合画像のファウショットプロンプトで推論と説明が強化される。
• GPT-4Vは世界知識問題で大きな幻覚を示し、知識の grounding の改善が必要である。
• GPT-4Vは複合画像で良好に動作するが、視覚を誤解したり視覚的手掛かりに過度に依存することがあり、いくつかのカテゴリで回答に影響を及ぼす。
• オープンソースのMLMsは多くの知識集約型VQAタスクでGPT-4Vに遅れを取り、カテゴリ間で長尾の性能ギャップが目立つ。
• ファウショットプロンプトは一部のドメインでGPT-4Vの性能を向上させるが、モデルとカテゴリによって効果が異なる。