[Paper Review] Kleister: A novel task for Information Extraction involving Long Documents with Complex Layout

Filip Graliński, Tomasz Stanisławek+6 more|arXiv (Cornell University)|Mar 4, 2020

Topic Modeling14 references22 citations

TL;DR

This paper introduces Kleister, a novel information extraction task designed for long, complex-layout documents such as contracts and charity reports, combining textual and structural layout features. It proposes two new datasets (Kleister-NDA and Kleister-Charity), evaluates text-only baselines using NLP models (Flair, BERT, RoBERTa), and demonstrates that OCR noise and layout complexity significantly degrade performance, highlighting the need for multimodal approaches beyond pure text processing.

ABSTRACT

State-of-the-art solutions for Natural Language Processing (NLP) are able to capture a broad range of contexts, like the sentence-level context or document-level context for short documents. But these solutions are still struggling when it comes to longer, real-world documents with the information encoded in the spatial structure of the document, such as page elements like tables, forms, headers, openings or footers; complex page layout or presence of multiple pages. To encourage progress on deeper and more complex Information Extraction (IE) we introduce a new task (named Kleister) with two new datasets. Utilizing both textual and structural layout features, an NLP system must find the most important information, about various types of entities, in long formal documents. We propose Pipeline method as a text-only baseline with different Named Entity Recognition architectures (Flair, BERT, RoBERTa). Moreover, we checked the most popular PDF processing tools for text extraction (pdf2djvu, Tesseract and Textract) in order to analyze behavior of IE system in presence of errors introduced by these tools.

Motivation & Objective

• To address the gap in existing NLP systems for extracting structured information from long, real-world documents with complex layouts, such as contracts and reports.
• To develop a new benchmark task—Kleister—that captures real-world challenges including layout complexity, OCR noise, document-level reasoning, and normalization.
• To evaluate the limitations of text-only NLP models in handling layout-aware and OCR-robust information extraction.
• To provide publicly available datasets (Kleister-NDA and Kleister-Charity) with detailed annotations for training and evaluating robust IE systems.
• To demonstrate that current state-of-the-art NLP models fail to generalize when faced with layout and OCR artifacts, even when using advanced architectures like BERT and RoBERTa.

Proposed method

• Propose a new task, Kleister, focused on extracting structured entities (e.g., dates, addresses, organizations) from long documents with complex layouts such as tables, forms, headers, and footers.
• Create two new datasets: Kleister-NDA (200 legal contracts) and Kleister-Charity (200 charity reports), annotated with entity types and normalized values.
• Use a Pipeline method as a text-only baseline, applying named entity recognition models (Flair, BERT, RoBERTa) on text extracted via PDF tools (Tesseract, Textract, pdf2djvu).
• Evaluate performance across different PDF extraction tools to assess the impact of OCR errors on downstream IE tasks.
• Incorporate document-level context, business logic (e.g., resolving ambiguous entity references), and normalization rules (e.g., date formats, 'Ltd' vs 'Limited') in the annotation and evaluation process.
• Analyze layout complexity by categorizing documents into simple, two-column, email, plain text, and form/report layouts with graphics and tables.

Experimental results

Research questions

• RQ1 How do state-of-the-art text-only NLP models perform on long, complex-layout documents with OCR artifacts and layout-dependent entity relationships?
• RQ2 To what extent do common PDF extraction tools (Tesseract, Textract, pdf2djvu) degrade the quality of text input and affect downstream IE performance?
• RQ3 How do layout complexity and OCR noise impact the accuracy of named entity recognition and normalization in real-world business documents?
• RQ4 What are the key challenges in document-level reasoning, such as disambiguating multiple addresses or interpreting co-occurring time periods in contracts?
• RQ5 Can a text-only baseline model achieve robust performance on a task requiring spatial and structural understanding beyond sentence-level context?

Key findings

• Text-only NLP models (Flair, BERT, RoBERTa) achieve limited performance on the Kleister task, indicating that reliance on pure text input is insufficient for complex document understanding.
• OCR quality significantly degrades performance, especially in documents with handwriting, poor scans, or upside-down pages, as seen in the Kleister-Charity dataset.
• Layout complexity—such as multi-column layouts, tables, and forms—introduces substantial challenges that text-only models fail to resolve, even with advanced contextual encoders.
• Normalization issues, such as variations in date formats (e.g., 'October 24, 2012' vs '10/24/12') or organization name variants (e.g., 'Ltd' vs 'Limited'), reduce model accuracy and require explicit handling.
• The Pipeline baseline method shows that performance varies significantly depending on the PDF extraction tool used, with Tesseract and Textract introducing more errors than pdf2djvu in layout-sensitive contexts.
• Document-level reasoning, such as identifying the correct address among multiple options (e.g., registered vs principal office), is poorly handled by text-only models, underscoring the need for layout-aware modeling.