[Paper Review] DocTr: Document Transformer for Structured Information Extraction in Documents

Haofu Liao, Aruni RoyChowdhury+7 more|arXiv (Cornell University)|Jul 16, 2023

Natural Language Processing Techniques4 citations

TL;DR

This paper proposes DocTr, a document transformer model that formulates structured information extraction (SIE) as an anchor-based object detection task, representing entities via anchor words and bounding boxes to reduce reliance on text order and simplify entity linking. The method outperforms existing IOB and graph-based approaches on three SIE benchmarks, with a 94.4% F1 on CORD receipt parsing using a novel masked detection modeling pre-training task.

ABSTRACT

We present a new formulation for structured information extraction (SIE) from visually rich documents. It aims to address the limitations of existing IOB tagging or graph-based formulations, which are either overly reliant on the correct ordering of input text or struggle with decoding a complex graph. Instead, motivated by anchor-based object detectors in vision, we represent an entity as an anchor word and a bounding box, and represent entity linking as the association between anchor words. This is more robust to text ordering, and maintains a compact graph for entity linking. The formulation motivates us to introduce 1) a DOCument TRansformer (DocTr) that aims at detecting and associating entity bounding boxes in visually rich documents, and 2) a simple pre-training strategy that helps learn entity detection in the context of language. Evaluations on three SIE benchmarks show the effectiveness of the proposed formulation, and the overall approach outperforms existing solutions.

Motivation & Objective

• To address the limitations of IOB tagging and graph-based methods in structured information extraction from visually rich documents.
• To reduce dependency on correct text reading order by modeling entities as anchor words and bounding boxes.
• To simplify entity linking through a compact graph of anchor word associations.
• To improve document understanding via a new pre-training task tailored to bounding box and text prediction.
• To develop a unified vision-language model that jointly detects entities and links them using language-conditioned queries.

Proposed method

• Formulate SIE as an anchor-based detection problem: each entity is represented by an anchor word and a bounding box, with entity linking modeled as associations between anchor words.
• Introduce DocTr, a vision-language transformer that combines a vision encoder, language encoder, and a vision-language decoder with language-conditioned queries (LCQ) for one-to-one OCR-to-output mapping.
• Propose masked detection modeling (MDM), a pre-training task that masks both OCR text and bounding boxes, requiring the model to predict both the masked word and its location.
• Use a multi-task loss combining detection loss (for bounding boxes) and language modeling loss (for text) during pre-training and fine-tuning.
• Apply language-conditioned queries (LCQ) in cross-attention layers to guide the model in aligning OCR tokens with predicted entity boxes.
• Fine-tune the model on downstream SIE tasks using standard detection and classification heads.

Experimental results

Research questions

• RQ1 Can a detection-based formulation for SIE reduce dependency on input text order compared to IOB tagging?
• RQ2 Does modeling entity linking via anchor word associations improve decoding robustness compared to complex graph-based methods?
• RQ3 Can a pre-training task that jointly masks text and bounding boxes (MDM) improve entity detection and localization in documents?
• RQ4 How effective is the integration of vision and language modeling in a unified transformer architecture for document understanding?
• RQ5 To what extent does using language-conditioned queries enhance the alignment between OCR tokens and predicted entity boxes?

Key findings

• The proposed anchor-based formulation outperforms IOB tagging and graph-based methods on all three SIE benchmarks, with a 94.4% F1 score on the CORD receipt parsing task.
• The masked detection modeling (MDM) pre-training task significantly improves performance, outperforming both no pre-training and MVLM pre-training, especially in bounding box prediction.
• The full DocTr model with vision encoder, language encoder, and VL-decoder achieves 94.4% F1 on CORD, a 4.1% improvement over MVLM pre-training.
• Ablation studies show that the vision encoder and language-conditioned queries (LCQ) are critical, with LCQ improving performance by 1.3% on ELB and 1.1% on ELK.
• The model generalizes well across document types, with strong performance on FUNSD (84.0% F1 on ELB) and CORD (94.4% F1 on parsing), even when anchor words vary per entity type.
• Example predictions show the model leverages visual context to predict masked words and boxes accurately, including non-textual values like prices.