[Paper Review] VisualMRC: Machine Reading Comprehension on Document Images

Ryota Tanaka, Kyosuke Nishida+1 more|arXiv (Cornell University)|Jan 27, 2021

Multimodal Machine Learning Applications47 references4 citations

TL;DR

This paper introduces VisualMRC, a new dataset and model for visual machine reading comprehension on document images, where models must understand layout, text, and visual content to generate abstractive answers. The proposed LayoutT5 model, fine-tuned from T5 with visual layout and saliency features, outperforms base sequence-to-sequence models and a state-of-the-art VQA model on the dataset, though human performance remains superior.

ABSTRACT

Recent studies on machine reading comprehension have focused on text-level understanding but have not yet reached the level of human understanding of the visual layout and content of real-world documents. In this study, we introduce a new visual machine reading comprehension dataset, named VisualMRC, wherein given a question and a document image, a machine reads and comprehends texts in the image to answer the question in natural language. Compared with existing visual question answering (VQA) datasets that contain texts in images, VisualMRC focuses more on developing natural language understanding and generation abilities. It contains 30,000+ pairs of a question and an abstractive answer for 10,000+ document images sourced from multiple domains of webpages. We also introduce a new model that extends existing sequence-to-sequence models, pre-trained with large-scale text corpora, to take into account the visual layout and content of documents. Experiments with VisualMRC show that this model outperformed the base sequence-to-sequence models and a state-of-the-art VQA model. However, its performance is still below that of humans on most automatic evaluation metrics. The dataset will facilitate research aimed at connecting vision and language understanding.

Motivation & Objective

• To address the gap in machine reading comprehension for real-world document images that include complex layouts and visual content.
• To develop a dataset that emphasizes natural language understanding and generation on document images, rather than simple visual grounding.
• To enable models to reason over multiple text and visual elements in document images to produce abstractive answers.
• To transfer pre-trained sequence-to-sequence models' language generation capabilities to visual reading comprehension without catastrophic forgetting.
• To advance vision-and-language models toward human-level comprehension of real-world digital documents.

Proposed method

• The authors created VisualMRC, a dataset of 10,000+ document images from diverse web domains, each paired with 30,000+ question-answer pairs requiring abstractive reasoning.
• The dataset includes regions-of-interest (ROIs) annotated with semantic classes (e.g., list, table, chart), enabling layout-aware understanding.
• A novel model, LayoutT5, is proposed by extending the T5 encoder-decoder architecture with visual layout embeddings and an auxiliary saliency detection head.
• The model is pre-trained on large-scale text corpora and fine-tuned on VisualMRC using both question-answer pairs and visual layout features.
• Visual features are extracted from document images using a Faster R-CNN detector to localize and classify ROIs, which are then encoded with positional and layout embeddings.
• The model uses a multi-task learning objective combining answer generation and saliency prediction to preserve language generation capacity while integrating visual information.

Experimental results

Research questions

• RQ1 Can pre-trained sequence-to-sequence models be effectively adapted for visual machine reading comprehension on document images while preserving their natural language generation ability?
• RQ2 How does incorporating visual layout and content information improve performance on document-level question answering compared to purely textual models?
• RQ3 To what extent can a vision-and-language model trained on text-to-text tasks generalize to document images with complex layouts and mixed visual elements?
• RQ4 How does the proposed LayoutT5 model compare to state-of-the-art VQA models in understanding and generating answers for document images?
• RQ5 What are the key failure modes in visual document comprehension, particularly regarding diagrams and visual objects?

Key findings

• The LayoutT5 model outperformed both the base T5 and BART models fine-tuned only on textual inputs, demonstrating the benefit of visual layout integration.
• LayoutT5 achieved better performance than M4C, a state-of-the-art vision-and-language model for VQA, on the VisualMRC dataset, indicating its superiority for abstractive document QA.
• The model's performance on questions involving diagrams or visual attributes (e.g., color of an object) remained limited, revealing a key failure mode in visual object understanding.
• The abstractive answer generation task in VisualMRC requires more complex reasoning than existing VQA datasets, as evidenced by the high human performance gap on automatic metrics.
• The auxiliary saliency detection task helped prevent catastrophic forgetting during visual fine-tuning, preserving the pre-trained model's language generation quality.
• The dataset's diversity in document types and the inclusion of long, abstractive answers make it suitable for evaluating advanced NLU and NLG capabilities on real-world documents.