[Paper Review] LAMBERT: Layout-Aware language Modeling using BERT for information extraction.

Łukasz Garncarek, Rafał Powalski+4 more|arXiv (Cornell University)|Feb 19, 2020

Natural Language Processing Techniques25 citations

TL;DR

LAMBERT introduces a layout-aware BERT-based model that integrates document layout features into pre-trained language representations without retraining from scratch. By fusing spatial and textual features via a modified Transformer, it achieves state-of-the-art performance in document understanding tasks, particularly in scenarios where layout significantly influences semantic interpretation.

ABSTRACT

In this paper we introduce a novel approach to the problem of understanding documents where the local semantics is influenced by non-trivial layout. Namely, we modify the Transformer architecture in a way that allows it to use the graphical features defined by the layout, without the need to re-learn the language semantics from scratch, thanks to starting the training process from a model pretrained on classical language modeling tasks.

Motivation & Objective

• To address the challenge of document understanding when layout features significantly influence local semantics.
• To integrate graphical layout information (e.g., position, size, proximity) into a pre-trained language model like BERT.
• To preserve the pre-trained language representations while adapting them to layout-aware reasoning.
• To improve performance in information extraction tasks on layout-sensitive documents such as forms and invoices.

Proposed method

• Extends the BERT architecture by introducing layout-conditional attention mechanisms that incorporate spatial coordinates into self-attention computation.
• Uses bounding box coordinates (x, y, width, height) of tokens as additional input embeddings alongside wordpiece embeddings.
• Introduces a learnable layout embedding that is concatenated with token embeddings to condition attention on spatial context.
• Fine-tunes the model end-to-end on document-level information extraction tasks, preserving the pre-trained language understanding capacity.
• Applies a multi-modal fusion strategy that jointly optimizes for both textual and spatial features during fine-tuning.

Experimental results

Research questions

• RQ1 Can layout features be effectively integrated into a pre-trained language model without retraining the language component from scratch?
• RQ2 How does the inclusion of spatial layout information improve document understanding in layout-sensitive scenarios?
• RQ3 To what extent does the proposed layout-aware attention mechanism outperform standard BERT in document-level information extraction tasks?
• RQ4 Does the model generalize across diverse document types such as forms, invoices, and scientific papers?

Key findings

• LAMBERT achieves state-of-the-art performance on multiple document-level information extraction benchmarks, including FUNSD and SROIE.
• The integration of layout features leads to a significant improvement in F1-score over standard BERT, particularly on complex layouts with non-linear structures.
• The model maintains strong language understanding capabilities due to the use of pre-trained weights, avoiding catastrophic forgetting.
• Ablation studies confirm that layout embeddings and spatial attention are critical components for performance gains.