[Paper Review] LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding

Yiheng Xu, Tengchao Lv+6 more|arXiv (Cornell University)|Apr 18, 2021

Natural Language Processing Techniques21 references48 citations

TL;DR

LayoutXLM is a multimodal pre-trained model for multilingual visually-rich document understanding, introducing XFUND for form understanding and achieving state-of-the-art results on multilingual benchmarks.

ABSTRACT

Multimodal pre-training with text, layout, and image has achieved SOTA performance for visually-rich document understanding tasks recently, which demonstrates the great potential for joint learning across different modalities. In this paper, we present LayoutXLM, a multimodal pre-trained model for multilingual document understanding, which aims to bridge the language barriers for visually-rich document understanding. To accurately evaluate LayoutXLM, we also introduce a multilingual form understanding benchmark dataset named XFUND, which includes form understanding samples in 7 languages (Chinese, Japanese, Spanish, French, Italian, German, Portuguese), and key-value pairs are manually labeled for each language. Experiment results show that the LayoutXLM model has significantly outperformed the existing SOTA cross-lingual pre-trained models on the XFUND dataset. The pre-trained LayoutXLM model and the XFUND dataset are publicly available at https://aka.ms/layoutxlm.

Motivation & Objective

• Bridge language barriers in Visually-rich Document Understanding (VrDU) by leveraging text, layout, and image modalities.
• Pre-train a multilingual model on large-scale real-world documents to capture cross-lingual and cross-modal signals.
• Create and release XFUND, a multilingual form understanding dataset with key-value annotations in 7 languages.
• Demonstrate that multimodal pre-training improves cross-lingual transfer and multitask performance on form understanding tasks.

Proposed method

• Adopts a multimodal Transformer architecture that encodes text, layout, and image features with text, layout, and visual embeddings.
• Extends LayoutLMv2 with multilingual pre-training initialized from InfoXLM and a visual backbone (ResNeXt101-FPN).
• Uses three pre-training objectives: Multilingual Masked Visual-Language Modeling (MMVLM), Text-Image Alignment (TIA), and Text-Image Matching (TIM).
• For MMVLM, token boundaries are defined at the character level with SentencePiece, unifying multilingual inputs via bounding-box aggregation.
• Training data comprises 30 million documents (30M total: 22M multilingual digital-born PDFs and 8M scanned English IIT-CDIP documents).

Experimental results

Research questions

• RQ1 Can a multimodal multilingual model effectively learn cross-lingual and cross-domain representations for visually-rich documents?
• RQ2 Does incorporating layout and image information improve form understanding across languages beyond text-only models?
• RQ3 How well does cross-lingual transfer work for multilingual form understanding under zero-shot and multitask settings?

Key findings

• LayoutXLM LARGE achieves the highest F1 across SER and RE on XFUND versus XLM-RoBERTa and InfoXLM baselines.
• In language-specific fine-tuning, LayoutXLM LARGE outperforms baselines on SER and RE across all eight language settings.
• In zero-shot transfer (train on English FUNSD, test on others), LayoutXLM significantly outperforms text-based models by leveraging layout invariance.
• Multitask fine-tuning across eight languages further improves performance compared to language-specific fine-tuning, demonstrating benefits of multilingual learning.