[Paper Review] LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding

Yang Xu, Yiheng Xu+10 more|arXiv (Cornell University)|Dec 29, 2020

Multimodal Machine Learning Applications42 references58 citations

TL;DR

LayoutLMv2 introduces a multi-modal Transformer that jointly models text, layout, and image for Visually-Rich Document Understanding, using new pre-training tasks and spatial-aware attention to achieve state-of-the-art results across multiple VrDU tasks.

ABSTRACT

Pre-training of text and layout has proved effective in a variety of visually-rich document understanding tasks due to its effective model architecture and the advantage of large-scale unlabeled scanned/digital-born documents. We propose LayoutLMv2 architecture with new pre-training tasks to model the interaction among text, layout, and image in a single multi-modal framework. Specifically, with a two-stream multi-modal Transformer encoder, LayoutLMv2 uses not only the existing masked visual-language modeling task but also the new text-image alignment and text-image matching tasks, which make it better capture the cross-modality interaction in the pre-training stage. Meanwhile, it also integrates a spatial-aware self-attention mechanism into the Transformer architecture so that the model can fully understand the relative positional relationship among different text blocks. Experiment results show that LayoutLMv2 outperforms LayoutLM by a large margin and achieves new state-of-the-art results on a wide variety of downstream visually-rich document understanding tasks, including FUNSD (0.7895 $ o$ 0.8420), CORD (0.9493 $ o$ 0.9601), SROIE (0.9524 $ o$ 0.9781), Kleister-NDA (0.8340 $ o$ 0.8520), RVL-CDIP (0.9443 $ o$ 0.9564), and DocVQA (0.7295 $ o$ 0.8672). We made our model and code publicly available at \url{https://aka.ms/layoutlmv2}.

Motivation & Objective

• Motivate improving VrDU by leveraging cross-modal information (text, layout, image) in a single pre-training framework.
• Propose a multi-modal Transformer with spatial-aware self-attention to learn cross-modal interactions and relative spatial relations.
• Introduce new pre-training tasks (text-image alignment and text-image matching) in addition to masked visual-language modeling.
• Evaluate on diverse VrDU datasets to demonstrate state-of-the-art performance and transferability to downstream tasks.

Proposed method

• Build a two-stream multi-modal Transformer encoder that fuses text embeddings, visual embeddings from a CNN backbone (ResNeXt-FPN), and layout embeddings from OCR bounding boxes.
• Incorporate spatial-aware self-attention with 1D and 2D relative position biases to model relative spatial relations among text blocks.
• Pre-training tasks include: Masked Visual-Language Modeling (MVLM), Text-Image Alignment (TIA) at line level, and Text-Image Matching (TIM).
• Use IIT-CDIP for pre-training and fine-tune on six downstream VrDU tasks including FUNSD, CORD, SROIE, Kleister-NDA, RVL-CDIP, and DocVQA.
• Fine-tune LayoutLMv2 variants (BASE, LARGE) on downstream tasks and compare against BERT, UniLMv2, LayoutLM (BASE/LARGE).

Experimental results

Research questions

• RQ1 How can cross-modal (text, layout, image) information be effectively modeled in a single pre-training framework for VrDU?
• RQ2 Do new pre-training objectives (TIA, TIM) and spatial-aware attention improve cross-modal understanding beyond MVLM?
• RQ3 What performance gains can be achieved on diverse VrDU tasks by LayoutLMv2 compared to prior models?
• RQ4 How does LayoutLMv2 transfer to both information extraction and document QA tasks?

Key findings

• LayoutLMv2 LARGE achieves state-of-the-art results across multiple VrDU tasks compared to prior models.
• On FUNSD, CORD, SROIE, and Kleister-NDA, LayoutLMv2 LARGE reaches 0.8420, 0.9601, 0.9781, and 0.8520 respectively (entity extraction F1).
• LayoutLMv2 BASE/LARGE outperform LayoutLM BASE/LARGE and text-only baselines, highlighting the benefit of multimodal pre-training.
• Text-Image Alignment (TIA) and Text-Image Matching (TIM) tasks, together with spatial-aware self-attention, significantly boost performance over MVLM alone.
• LayoutLMv2 also improves document image classification (RVL-CDIP) and DocVQA ANLS scores, indicating broad benefit to VrDU tasks.
• The model and code are publicly available for replication and extension.