[Paper Review] LAMBERT: Layout-Aware (Language) Modeling for information extraction

Łukasz Garncarek, Rafał Powalski+5 more|arXiv (Cornell University)|Feb 19, 2020

Handwritten Text Recognition TechniquesComputer Science34 references87 citations

TL;DR

LAMBERT is a layout-aware language model that enhances RoBERTa by injecting token bounding box coordinates into the input embeddings, enabling contextual understanding of document layout without using raw images. It achieves state-of-the-art performance on key information extraction tasks, ranking first on the SROIE leaderboard with a 98.17 F1-score, and improves baseline RoBERTa by 1.92 F1 points on the Kleister NDA dataset.

ABSTRACT

We introduce a simple new approach to the problem of understanding documents where non-trivial layout influences the local semantics. To this end, we modify the Transformer encoder architecture in a way that allows it to use layout features obtained from an OCR system, without the need to re-learn language semantics from scratch. We only augment the input of the model with the coordinates of token bounding boxes, avoiding, in this way, the use of raw images. This leads to a layout-aware language model which can then be fine-tuned on downstream tasks. The model is evaluated on an end-to-end information extraction task using four publicly available datasets: Kleister NDA, Kleister Charity, SROIE and CORD. We show that our model achieves superior performance on datasets consisting of visually rich documents, while also outperforming the baseline RoBERTa on documents with flat layout (NDA \(F_{1}\) increase from 78.50 to 80.42). Our solution ranked first on the public leaderboard for the Key Information Extraction from the SROIE dataset, improving the SOTA \(F_{1}\)-score from 97.81 to 98.17.

Motivation & Objective

• To address the limitation of standard NLP models in capturing layout semantics in visually rich documents.
• To develop a simple, efficient method that integrates layout features into pretrained language models without retraining from scratch.
• To improve end-to-end information extraction on documents with complex layouts, such as forms, tables, and legal texts.
• To enable unsupervised pretraining on large-scale document collections while preserving performance on flat-text documents.
• To provide a plug-and-play solution compatible with existing OCR pipelines by using only bounding box coordinates.

Proposed method

• Augment RoBERTa’s input embeddings with 2D bounding box coordinates of tokens, preserving the original model architecture.
• Introduce relative 2D positional bias in the self-attention mechanism to model spatial relationships between tokens.
• Use an adapter layer to project layout embeddings into the model’s hidden dimension, enabling controlled fusion of layout and semantic features.
• Train the model in an unsupervised manner on a 2M-page dataset of visually rich documents, filtered for quality and diversity.
• Fine-tune the layout-augmented model on downstream information extraction tasks without requiring layout annotations.
• Evaluate ablation studies on layout embedding types, dimensionality, and training data size to isolate the impact of layout features.

Experimental results

Research questions

• RQ1 Can injecting bounding box coordinates into a pretrained RoBERTa model significantly improve performance on information extraction from visually complex documents?
• RQ2 How does the inclusion of layout features affect performance on documents with flat, linear layouts?
• RQ3 What is the optimal way to integrate layout information—via absolute layout embeddings, relative attention bias, or both—within a transformer-based architecture?
• RQ4 Does the performance of the layout-augmented model scale with larger, high-quality unsupervised pretraining datasets?
• RQ5 How does the adapter layer influence the model’s ability to balance semantic and layout representations?

Key findings

• LAMBERT achieved a state-of-the-art F1-score of 98.17 on the SROIE dataset, surpassing the previous SOTA of 97.81.
• On the Kleister NDA dataset, LAMBERT improved the F1-score from 78.50 (RoBERTa baseline) to 80.42, a 1.92-point gain.
• The model outperformed RoBERTa on all four evaluated datasets, including visually rich documents like forms and legal contracts.
• The ablation study showed that combining sequential and layout embeddings with relative 2D attention bias yielded the best results, with statistical significance on SROIE* and CORD.
• Using a larger, filtered pretraining dataset of 75M pages led to significant performance gains on Kleister Charity and SROIE*, indicating that data quality and scale are critical.
• The model with 768-dimensional layout embeddings and no adapter layer outperformed other configurations on Kleister Charity, suggesting that direct layout signal injection can be more effective when the adapter is not needed.