[Paper Review] MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answering

Jingqun Tang, Qi Liu+15 more|arXiv (Cornell University)|May 20, 2024

Multimodal Machine Learning Applications4 citations

TL;DR

MTVQA introduces the first multilingual, text-centric visual question answering benchmark with high-quality human expert annotations across nine diverse languages, addressing visual-textual misalignment in translation-based methods. Evaluation reveals significant performance gaps in state-of-the-art MLLMs, highlighting substantial room for improvement in multilingual text-rich visual understanding.

ABSTRACT

Text-Centric Visual Question Answering (TEC-VQA) in its proper format not only facilitates human-machine interaction in text-centric visual environments but also serves as a de facto gold proxy to evaluate AI models in the domain of text-centric scene understanding. Nonetheless, most existing TEC-VQA benchmarks have focused on high-resource languages like English and Chinese. Despite pioneering works to expand multilingual QA pairs in non-text-centric VQA datasets through translation engines, the translation-based protocol encounters a substantial "visual-textual misalignment" problem when applied to TEC-VQA. Specifically, it prioritizes the text in question-answer pairs while disregarding the visual text present in images. Moreover, it fails to address complexities related to nuanced meaning, contextual distortion, language bias, and question-type diversity. In this work, we tackle multilingual TEC-VQA by introducing MTVQA, the first benchmark featuring high-quality human expert annotations across 9 diverse languages, consisting of 6,778 question-answer pairs across 2,116 images. Further, by comprehensively evaluating numerous state-of-the-art Multimodal Large Language Models~(MLLMs), including Qwen2-VL, GPT-4o, GPT-4V, Claude3, and Gemini, on the MTVQA benchmark, it is evident that there is still a large room for performance improvement (Qwen2-VL scoring 30.9 versus 79.7 for human performance), underscoring the value of MTVQA. Additionally, we supply multilingual training data within the MTVQA dataset, demonstrating that straightforward fine-tuning with this data can substantially enhance multilingual TEC-VQA performance. We aspire that MTVQA will offer the research community fresh insights and stimulate further exploration in multilingual visual text comprehension. The project homepage is available at https://bytedance.github.io/MTVQA/.

Motivation & Objective

• Address the lack of high-quality, multilingual text-centric VQA benchmarks, especially for low-resource languages.
• Overcome the visual-textual misalignment problem inherent in translation-based data expansion for TEC-VQA.
• Provide a benchmark that captures nuanced linguistic challenges, contextual distortion, and question-type diversity in multilingual settings.
• Enable rigorous evaluation of multimodal LLMs in multilingual, text-rich visual environments.
• Inspire future research in multilingual, document-focused, and instruction-tuned MLLMs for text-centric vision tasks.

Proposed method

• Collect real-world text-rich images (e.g., menus, bills, PPTs, research papers) from diverse sources to ensure visual-textual relevance.
• Employ a two-stage human annotation pipeline: first, annotators generate questions and answers; second, a separate group verifies accuracy and consistency.
• Focus on nine languages: Arabic, Korean, Japanese, Thai, Vietnamese, Russian, French, German, and Italian, ensuring linguistic diversity.
• Ensure visual-textual alignment by requiring annotators to consider both image content and textual elements when formulating questions.
• Design the dataset with fine-grained scenarios to cover content extraction, reasoning, and contextual understanding across languages.
• Provide a canonical answer per question, with plans to expand to multiple plausible answers in future versions.

Experimental results

Research questions

• RQ1 Can a multilingual TEC-VQA benchmark with human expert annotations overcome the visual-textual misalignment problem seen in translation-based data expansion?
• RQ2 How do state-of-the-art MLLMs perform on multilingual, text-rich visual question answering compared to monolingual benchmarks?
• RQ3 To what extent does instruction tuning improve MLLM performance on multilingual TEC-VQA tasks?
• RQ4 What are the performance disparities across languages, especially between high-resource and low-resource language groups?
• RQ5 How do document-focused MLLMs compare to general-purpose MLLMs on multilingual text-centric VQA?

Key findings

• Claude3 Opus achieved the highest average accuracy of 25.7% across all nine languages, indicating that multilingual TEC-VQA remains a significant challenge even for state-of-the-art models.
• Closed-source models, particularly GPT-4V and Gemini Ultra, outperformed open-source models overall, with GPT-4V achieving 22.0% average accuracy.
• Performance varied significantly by language, with Indo-European languages using the Latin script (e.g., French, German, Italian) showing higher accuracy than non-Latin scripts like Arabic, Thai, and Vietnamese.
• Instruction tuning improved average accuracy by 8.5 percentage points, with French showing the largest gain (14.2%) and Russian the smallest (1.7%).
• Document-focused models like TextSquare and TextMonkey did not significantly outperform general-purpose open-source models on the MTVQA benchmark.
• GLM4V showed particularly poor performance (13.6% average accuracy), with near-zero scores in Arabic and Korean, highlighting model-specific limitations in low-resource language understanding.