[Paper Review] Document AI: Benchmarks, Models and Applications

Lei Cui, Yiheng Xu+2 more|arXiv (Cornell University)|Nov 16, 2021

Handwritten Text Recognition Techniques101 references34 citations

TL;DR

The paper surveys Document AI, reviewing tasks, benchmark datasets, representative models, and pre-training approaches, with a focus on multimodal layouts and visual information, and discusses future directions.

ABSTRACT

Document AI, or Document Intelligence, is a relatively new research topic that refers to the techniques for automatically reading, understanding, and analyzing business documents. It is an important research direction for natural language processing and computer vision. In recent years, the popularity of deep learning technology has greatly advanced the development of Document AI, such as document layout analysis, visual information extraction, document visual question answering, document image classification, etc. This paper briefly reviews some of the representative models, tasks, and benchmark datasets. Furthermore, we also introduce early-stage heuristic rule-based document analysis, statistical machine learning algorithms, and deep learning approaches especially pre-training methods. Finally, we look into future directions for Document AI research.

Motivation & Objective

• Motivate the study of Document AI by highlighting its industrial importance across finance, healthcare, and logistics.
• Summarize representative tasks, datasets, and model families in Document AI, from heuristic and classical ML to deep learning.
• Analyze the role of pre-training and multimodal information (layout and visuals) in improving downstream tasks.
• Identify current challenges and propose directions for future research in Document AI.

Proposed method

• Classify and summarize historical and modern approaches from heuristic rule-based methods to statistical ML and deep learning.
• Discuss task-specific models for document layout analysis, visual information extraction, and document VQA.
• Introduce general-purpose multimodal pre-training architectures (e.g., LayoutLM) that integrate 2-D layout and image features with text.
• Describe self-supervised pre-training tasks (masked visual-language modeling) and optional multi-label document classification losses for pre-training.
• Summarize key benchmark datasets and their role in advancing document understanding tasks.

Experimental results

Research questions

• RQ1 What are the main tasks and benchmarks that define Document AI today?
• RQ2 How have models evolved from heuristic and classical ML to deep learning and multimodal pre-training in Document AI?
• RQ3 What are the current challenges in applying Document AI to real-world, long, and varied documents?
• RQ4 How do multimodal pre-training approaches like LayoutLM improve downstream Document AI tasks?
• RQ5 What datasets drive progress, and what future directions do the authors identify?

Key findings

• Document AI tasks include layout analysis, visual information extraction, document VQA, and image classification, supported by a growing set of benchmark datasets.
• CNN-based layout analysis, GNNs for visually rich documents, and Transformer-based multimodal pre-training (LayoutLM) significantly advance performance on downstream tasks.
• LayoutLM integrates 2-D layout and image embeddings with text in a Transformer framework, enabling effective knowledge transfer to downstream tasks.
• Pre-training with layout and visual information yields noticeable improvements on multiple downstream tasks, validating the multimodal pre-training approach.
• Challenges remain for long documents, cross-page understanding, data quality gaps from scanning, and the need for multi-task learning and resource-efficient models.