[Paper Review] Kleister: Key Information Extraction Datasets Involving Long Documents with Complex Layouts

Tomasz Stanisławek, Filip Graliński+6 more|arXiv (Cornell University)|May 12, 2021

Advanced Text Analysis TechniquesComputer Science22 references63 citations

TL;DR

This paper introduces Kleister NDA and Kleister Charity, two new datasets for Key Information Extraction (KIE) from long, formally structured documents with complex layouts. The datasets comprise 540 NDA contracts and 2,788 charity reports, totaling 64,872 pages and 23,772 entities. The best-performing model achieved 81.77% and 83.57% F1-score on the respective datasets, highlighting the challenge of layout-aware, document-level information extraction beyond standard NER tasks.

ABSTRACT

The relevance of the Key Information Extraction (KIE) task is increasingly important in natural language processing problems. But there are still only a few well-defined problems that serve as benchmarks for solutions in this area. To bridge this gap, we introduce two new datasets (Kleister NDA and Kleister Charity). They involve a mix of scanned and born-digital long formal English-language documents. In these datasets, an NLP system is expected to find or infer various types of entities by employing both textual and structural layout features. The Kleister Charity dataset consists of 2,788 annual financial reports of charity organizations, with 61,643 unique pages and 21,612 entities to extract. The Kleister NDA dataset has 540 Non-disclosure Agreements, with 3,229 unique pages and 2,160 entities to extract. We provide several state-of-the-art baseline systems from the KIE domain (Flair, BERT, RoBERTa, LayoutLM, LAMBERT), which show that our datasets pose a strong challenge to existing models. The best model achieved an 81.77% and an 83.57% F1-score on respectively the Kleister NDA and the Kleister Charity datasets. We share the datasets to encourage progress on more in-depth and complex information extraction tasks.

Motivation & Objective

• To address the lack of large-scale, realistic benchmarks for Key Information Extraction (KIE) in long, formally structured documents with complex layouts.
• To bridge the gap between existing public KIE datasets and real-world business document processing needs, particularly for contracts and financial reports.
• To evaluate state-of-the-art NLP models on document-level, layout-aware information extraction under realistic conditions including OCR noise and document length.
• To provide a semi-supervised data collection methodology that reduces manual annotation effort while maintaining high-quality entity annotations.

Proposed method

• The datasets were collected using a semi-supervised methodology combining automated document harvesting from public repositories with targeted human annotation for key entities.
• For the Kleister NDA dataset, born-digital PDFs were processed using specialized tools like pdf2djvu to preserve layout structure, while the charity dataset required OCR due to scanned inputs.
• Multiple PDF processing tools were evaluated—Azure Cognitive Services, Tesseract, Textract, and pdf2djvu—each impacting model performance significantly.
• Baseline models including Flair, BERT, RoBERTa, LayoutLM, and LAMBERT were trained and evaluated on both datasets using both text-only and layout-aware features.
• An autotagging mechanism based on regular expressions was used to generate training data, later refined with human-annotated spans for model training.
• Normalization tasks were explicitly modeled, such as inferring monetary scales (e.g., thousand/million) from context, which proved critical for financial entities.

Experimental results

Research questions

• RQ1 How do document length and layout complexity affect the performance of state-of-the-art KIE models?
• RQ2 To what extent does OCR quality and PDF processing pipeline choice impact model performance on real-world KIE tasks?
• RQ3 Can layout-aware models like LayoutLM and LAMBERT outperform standard sequence models in complex document understanding?
• RQ4 How do normalization tasks (e.g., inferring monetary units) affect KIE performance and what is their impact on model design?
• RQ5 What is the relative contribution of textual, structural, and layout features in extracting key entities from long, formally structured documents?

Key findings

• The best model, LAMBERT, achieved an 83.57% F1-score on the Kleister Charity dataset and 81.77% on the Kleister NDA dataset, significantly lower than performance on simpler datasets like SROIE.
• Layout-aware models (e.g., LAMBERT, LayoutLM) outperformed standard sequence models (Flair, BERT, RoBERTa), demonstrating the importance of 2D layout features.
• OCR quality had a substantial impact: commercial tools like Azure Cognitive Services and Textract outperformed Tesseract, especially for 1D models not trained on Tesseract output.
• Longer documents consistently led to lower F1-scores across all models, indicating a performance drop with document length.
• Normalization tasks—such as inferring monetary scales—were critical and often required context-aware reasoning, which even the best models struggled with.
• The autotagging mechanism, while less accurate than human annotation, enabled effective pre-training and showed that rule-based heuristics alone are insufficient for complex entities like abbreviated organization names.