[Paper Review] Robust Layout-aware IE for Visually Rich Documents with Pre-trained Language Models

Mengxi Wei, Yifan He+1 more|arXiv (Cornell University)|May 22, 2020

Multimodal Machine Learning Applications36 references4 citations

TL;DR

This paper proposes a graph neural network-enhanced RoBERTa model for robust, layout-aware information extraction from visually rich documents (VRDs), integrating text, layout, font, and structural features. It achieves 6.3% and 4.7% absolute F1 gains on invoice and resume datasets, respectively, and reduces annotation needs by up to 30× in few-shot settings via novel in-domain fine-tuning objectives.

ABSTRACT

Many business documents processed in modern NLP and IR pipelines are visually rich: in addition to text, their semantics can also be captured by visual traits such as layout, format, and fonts. We study the problem of information extraction from visually rich documents (VRDs) and present a model that combines the power of large pre-trained language models and graph neural networks to efficiently encode both textual and visual information in business documents. We further introduce new fine-tuning objectives to improve in-domain unsupervised fine-tuning to better utilize large amount of unlabeled in-domain data. We experiment on real world invoice and resume data sets and show that the proposed method outperforms strong text-based RoBERTa baselines by 6.3% absolute F1 on invoices and 4.7% absolute F1 on resumes. When evaluated in a few-shot setting, our method requires up to 30x less annotation data than the baseline to achieve the same level of performance at ~90% F1.

Motivation & Objective

• To improve information extraction (IE) in visually rich documents (VRDs) such as invoices and resumes by leveraging visual layout, font, and structural cues beyond plain text.
• To address the limitation of text-only models that ignore layout semantics, which hinders performance on domain-specific, visually complex documents.
• To reduce dependency on large-scale labeled data by introducing layout-aware in-domain fine-tuning objectives using unlabeled in-domain data.
• To enable robust few-shot learning by improving model generalization through better layout encoding and unsupervised pre-training.
• To model diverse layout semantics—such as section titles, proximity, and font variation—using a multi-relational graph neural network.

Proposed method

• Uses RoBERTa as the text encoder to capture deep contextual representations from document text.
• Constructs a heterogeneous graph where nodes represent text boxes and edges encode multiple layout relationships, including adjacency and section-title connections.
• Incorporates visual features such as bounding box coordinates, font size, and font type into node embeddings to enrich layout semantics.
• Employs a GCN-based message-passing mechanism that supports multiple edge types, allowing the model to learn from diverse layout patterns.
• Introduces two in-domain fine-tuning objectives: Sequence Positional Relationship Classification (SPRC) to learn layout relationships and Masked Language Modeling (MLM) to preserve semantic understanding.
• Applies SPRC with class-balanced sampling to handle skewed layout relationship distributions in resumes, improving generalization.

Experimental results

Research questions

• RQ1 Can integrating rich layout semantics—beyond bounding boxes—improve information extraction performance on visually rich documents?
• RQ2 How effective are graph neural networks in modeling complex layout relationships such as section titles and font-based formatting in documents?
• RQ3 To what extent can layout-aware in-domain fine-tuning reduce the need for labeled data in few-shot IE settings?
• RQ4 Does combining pre-trained language models with layout-aware graph encoders lead to better performance than text-only baselines on real-world business documents?
• RQ5 Which layout features—position, font, section structure—contribute most significantly to improved entity extraction accuracy?

Key findings

• The proposed model achieves 6.3% absolute F1 improvement over RoBERTa baselines on the invoice dataset, with a 4.7% gain on the resume dataset.
• On the resume dataset, the model reaches 72.13 F1 when fine-tuned with both MLM and SPRC objectives, outperforming single-objective or baseline methods.
• Ablation studies show that removing section title edges reduces F1 by 0.8 points, highlighting their importance in modeling document structure.
• Incorporating font features into node embeddings improves performance by 0.5 F1 points, demonstrating their value in distinguishing content types.
• Removing skip connections between GCN layers leads to a 0.4 F1 point drop, indicating that deep graph learning benefits from residual connections.
• In few-shot settings, the model achieves ~90% F1 with up to 30× less labeled data than the RoBERTa baseline, proving strong data efficiency.