[Paper Review] DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysis

B. Pfitzmann, Christoph Auer+3 more|arXiv (Cornell University)|Jun 2, 2022

Handwritten Text Recognition Techniques10 citations

TL;DR

This paper introduces DocLayNet, a large-scale, human-annotated dataset of 80,863 diverse document pages with 11 detailed layout class labels, designed to improve document-layout analysis. Trained on this dataset, object detection models achieve significantly more robust performance across varied layouts compared to models trained on PubLayNet or DocBank, with a 10% gap between model accuracy and human inter-annotator agreement indicating room for improvement.

ABSTRACT

Accurate document layout analysis is a key requirement for high-quality PDF document conversion. With the recent availability of public, large ground-truth datasets such as PubLayNet and DocBank, deep-learning models have proven to be very effective at layout detection and segmentation. While these datasets are of adequate size to train such models, they severely lack in layout variability since they are sourced from scientific article repositories such as PubMed and arXiv only. Consequently, the accuracy of the layout segmentation drops significantly when these models are applied on more challenging and diverse layouts. In this paper, we present extit{DocLayNet}, a new, publicly available, document-layout annotation dataset in COCO format. It contains 80863 manually annotated pages from diverse data sources to represent a wide variability in layouts. For each PDF page, the layout annotations provide labelled bounding-boxes with a choice of 11 distinct classes. DocLayNet also provides a subset of double- and triple-annotated pages to determine the inter-annotator agreement. In multiple experiments, we provide baseline accuracy scores (in mAP) for a set of popular object detection models. We also demonstrate that these models fall approximately 10\% behind the inter-annotator agreement. Furthermore, we provide evidence that DocLayNet is of sufficient size. Lastly, we compare models trained on PubLayNet, DocBank and DocLayNet, showing that layout predictions of the DocLayNet-trained models are more robust and thus the preferred choice for general-purpose document-layout analysis.

Motivation & Objective

• To address the limited layout variability in existing datasets like PubLayNet and DocBank, which are derived from scientific repositories with uniform templates.
• To provide a large-scale, human-annotated dataset with high diversity in document types and layouts to better train and evaluate layout segmentation models.
• To establish reliable baseline performance metrics for state-of-the-art object detection models on a diverse layout distribution.
• To quantify the gap between human annotation agreement and model performance, highlighting challenges in layout understanding.
• To demonstrate that models trained on DocLayNet generalize better across diverse document types than those trained on more homogeneous datasets.

Proposed method

• The dataset was constructed via a large-scale human annotation campaign, with 80,863 unique document pages annotated in COCO format using 11 distinct layout classes.
• A subset of pages (double- and triple-annotated) was used to compute inter-annotator agreement and assess annotation quality.
• The dataset includes predefined, non-overlapping train, validation, and test splits to prevent data leakage and ensure proportional class representation.
• Baseline models, including Mask R-CNN with ResNet-50, were trained and evaluated on DocLayNet, PubLayNet, and DocBank for comparative analysis.
• Ablation studies were conducted to evaluate the impact of data splitting strategies (page-wise vs. document-wise) on model performance metrics.
• Models were fine-tuned and evaluated on common label classes across datasets to compare generalization performance.

Experimental results

Research questions

• RQ1 How does the layout diversity in DocLayNet compare to existing datasets like PubLayNet and DocBank, and what impact does this have on model generalization?
• RQ2 What is the inter-annotator agreement on layout annotation in DocLayNet, and how does it compare to model prediction accuracy?
• RQ3 Does training on DocLayNet lead to better generalization performance on unseen, non-scientific document layouts compared to models trained on PubLayNet or DocBank?
• RQ4 How do different data splitting strategies (page-wise vs. document-wise) affect the evaluation of model mAP scores?
• RQ5 What is the performance gap between human annotators and machine learning models in document-layout analysis?

Key findings

• Models trained on DocLayNet achieve a mean average precision (mAP) of 78 on its own test set, significantly outperforming models trained on PubLayNet and DocBank when evaluated on the same set.
• The performance gap between DocLayNet-trained models and human inter-annotator agreement is approximately 10%, indicating that models still lag behind human-level understanding.
• Page-wise data splitting leads to a ~10% higher mAP score compared to document-wise splitting, highlighting the risk of overestimating model performance with improper splitting.
• Models trained on DocLayNet show more consistent performance across different datasets (PubLayNet, DocBank, and DocLayNet) than models trained on PubLayNet or DocBank, demonstrating superior generalization.
• The model trained on DocLayNet achieves an mAP of 84 on the Text class and 82 on the Table class in the DocLayNet test set, outperforming models trained on other datasets on these common labels.
• The study confirms that DocLayNet is sufficiently large and diverse to serve as a benchmark for future research in document-layout analysis.