[Paper Review] TRIE++: Towards End-to-End Information Extraction from Visually Rich Documents

Zhanzhan Cheng, Peng Zhang+7 more|arXiv (Cornell University)|Jul 14, 2022

Handwritten Text Recognition Techniques4 citations

TL;DR

This paper proposes TRIE++：a unified end-to-end framework for visually rich document (VRD) information extraction that jointly optimizes text reading and information extraction via a multi-modal context block. By fusing visual, textual, and layout features—along with pre-trained language model knowledge—it achieves state-of-the-art performance across diverse document types and layouts, significantly outperforming prior two-stage methods.

ABSTRACT

Recently, automatically extracting information from visually rich documents (e.g., tickets and resumes) has become a hot and vital research topic due to its widespread commercial value. Most existing methods divide this task into two subparts: the text reading part for obtaining the plain text from the original document images and the information extraction part for extracting key contents. These methods mainly focus on improving the second, while neglecting that the two parts are highly correlated. This paper proposes a unified end-to-end information extraction framework from visually rich documents, where text reading and information extraction can reinforce each other via a well-designed multi-modal context block. Specifically, the text reading part provides multi-modal features like visual, textual and layout features. The multi-modal context block is developed to fuse the generated multi-modal features and even the prior knowledge from the pre-trained language model for better semantic representation. The information extraction part is responsible for generating key contents with the fused context features. The framework can be trained in an end-to-end trainable manner, achieving global optimization. What is more, we define and group visually rich documents into four categories across two dimensions, the layout and text type. For each document category, we provide or recommend the corresponding benchmarks, experimental settings and strong baselines for remedying the problem that this research area lacks the uniform evaluation standard. Extensive experiments on four kinds of benchmarks (from fixed layout to variable layout, from full-structured text to semi-unstructured text) are reported, demonstrating the proposed method's effectiveness. Data, source code and models are available.

Motivation & Objective

• Address the limitation of existing two-stage VRD systems that treat text reading and information extraction as separate, non-interactive tasks.
• Overcome the lack of unified evaluation standards for visually rich documents by defining and grouping VRDs along layout and text type dimensions.
• Develop a unified, end-to-end trainable framework that allows mutual supervision between text reading and information extraction for global optimization.
• Provide standardized benchmarks and strong baselines for four document categories (fixed/variable layout, structured/semi-structured text) to promote reproducible research.
• Reduce dependency on off-the-shelf OCR engines by training a domain-specific, end-to-end system that improves robustness on real-world document variations.

Proposed method

• Integrate text reading and information extraction into a single end-to-end framework using a multi-modal context block to fuse visual, textual, and layout features.
• Extract multi-modal features from document images: visual features via CNN or ViT, textual features via transformer-based encoder, and layout features from bounding box coordinates.
• Inject pre-trained language model knowledge into the multi-modal context block to enhance semantic representation and contextual understanding.
• Enable end-to-end back-propagation by allowing the information extraction loss to supervise the text reading module, and vice versa, enabling mutual optimization.
• Use a shared encoder architecture to process document features and generate entity predictions with span-based head prediction.
• Train the entire system end-to-end with joint optimization, leveraging both detection and labeling supervision signals.

Experimental results

Research questions

• RQ1 Can joint optimization of text reading and information extraction through multi-modal feature fusion improve performance on visually rich documents?
• RQ2 How does the integration of visual, textual, and layout features—along with pre-trained language model knowledge—enhance semantic representation in document understanding?
• RQ3 To what extent does end-to-end training reduce error propagation and improve robustness on real-world document variations compared to two-stage pipelines?
• RQ4 How do different document categories (based on layout and text type) affect the performance of end-to-end information extraction systems?
• RQ5 Can a unified benchmarking framework with standardized evaluation protocols improve reproducibility and progress in VRD research?

Key findings

• TRIE++ achieves state-of-the-art performance across four benchmark categories, including fixed and variable layout documents with both structured and semi-structured text.
• The end-to-end training strategy significantly improves both text reading and information extraction performance, with the text recognition module emerging as a key bottleneck in Categories II, III, and IV.
• The multi-modal context block effectively fuses visual, textual, and layout features, leading to better entity recognition, especially for complex layouts and long text sequences.
• The method outperforms previous two-stage approaches by leveraging mutual supervision between text reading and information extraction, reducing error accumulation.
• Extensive ablation studies confirm that the integration of pre-trained language model knowledge and layout-aware features contributes significantly to performance gains.
• The proposed benchmarks and strong baselines provide a standardized evaluation platform, addressing the lack of uniform evaluation standards in VRD research.