[Paper Review] BERTgrid: Contextualized Embedding for 2D Document Representation and Understanding

Timo I. Denk, Christian Reisswig|arXiv (Cornell University)|Sep 11, 2019

Handwritten Text Recognition Techniques5 references69 citations

TL;DR

BERTgrid creates a 2D grid of contextualized BERT word-piece embeddings aligned to document layout, improving invoice field extraction over prior grid-based methods.

ABSTRACT

For understanding generic documents, information like font sizes, column layout, and generally the positioning of words may carry semantic information that is crucial for solving a downstream document intelligence task. Our novel BERTgrid, which is based on Chargrid by Katti et al. (2018), represents a document as a grid of contextualized word piece embedding vectors, thereby making its spatial structure and semantics accessible to the processing neural network. The contextualized embedding vectors are retrieved from a BERT language model. We use BERTgrid in combination with a fully convolutional network on a semantic instance segmentation task for extracting fields from invoices. We demonstrate its performance on tabulated line item and document header field extraction.

Motivation & Objective

• Leverage 2D document structure (layout, fonts, positions) to improve information extraction from invoices.
• Develop a grid-based representation that preserves spatial information while encoding semantic context.
• Evaluate the approach on header and line-item field extraction against existing grid-based methods.

Proposed method

• Represent documents as a 2D grid where each cell contains a contextualized BERT word-piece embedding for the word occupying that spatial region.
• Embed by feeding the serialized document text into a pre-trained BERT model and use the second-to-last hidden layer as the embedding for each word-piece.
• Combine OCR-extracted word positions with BERT embeddings to form the input tensor W used by a fully convolutional network for semantic segmentation and bounding-box regression.
• Compare BERTgrid to Chargrid, Wordgrid, and hybrid variants (C+BERTgrid, C+Wordgrid) using an identical downstream network to isolate input representation effects.
• Train on 800k iterations per model on an Nvidia V100, with BERT not fine-tuned on the downstream task.

Experimental results

Research questions

• RQ1 Does contextualized, word-piece level embedding in a 2D document grid improve extraction accuracy for invoice headers and line items compared to non-contextual or character-level grids?
• RQ2 How does BERTgrid perform relative to Chargrid and Wordgrid, and do combination models (C+BERTgrid, C+Wordgrid) offer additional gains?
• RQ3 What is the impact of using a domain-specific pre-trained BERT representation on downstream information extraction tasks in documents with complex layouts?

Key findings

• BERTgrid and its hybrid C+BERTgrid outperform all baselines on average extraction accuracy across header and line-item fields.
• C+BERTgrid achieves 65.48% mean extraction accuracy, 6.02% relative improvement over Chargrid baseline.
• Contextualized, word-piece embeddings provide faster convergence and better handling of layout-dependent semantics than non-contextual word/character grids.
• Wordgrid alone performs worse due to higher out-of-vocabulary issues, while combining word- and character-level information yields the best results.
• Pre-training BERT on a large unlabeled invoice corpus helps distill domain knowledge into embeddings used by the downstream model.