[Paper Review] OCR-free Document Understanding Transformer

Geewook Kim, Teakgyu Hong+8 more|arXiv (Cornell University)|Nov 30, 2021

Handwritten Text Recognition Techniques5 citations

TL;DR

This paper proposes Donut, an end-to-end OCR-free Visual Document Understanding (VDU) model based on a Transformer architecture that directly maps document images to structured outputs. By pre-training on synthetic data with a simple cross-entropy objective for text prediction, Donut achieves state-of-the-art accuracy and speed across multiple VDU tasks without relying on OCR, while enabling multi-lingual generalization.

ABSTRACT

Understanding document images (e.g., invoices) is a core but challenging task since it requires complex functions such as reading text and a holistic understanding of the document. Current Visual Document Understanding (VDU) methods outsource the task of reading text to off-the-shelf Optical Character Recognition (OCR) engines and focus on the understanding task with the OCR outputs. Although such OCR-based approaches have shown promising performance, they suffer from 1) high computational costs for using OCR; 2) inflexibility of OCR models on languages or types of document; 3) OCR error propagation to the subsequent process. To address these issues, in this paper, we introduce a novel OCR-free VDU model named Donut, which stands for Document understanding transformer. As the first step in OCR-free VDU research, we propose a simple architecture (i.e., Transformer) with a pre-training objective (i.e., cross-entropy loss). Donut is conceptually simple yet effective. Through extensive experiments and analyses, we show a simple OCR-free VDU model, Donut, achieves state-of-the-art performances on various VDU tasks in terms of both speed and accuracy. In addition, we offer a synthetic data generator that helps the model pre-training to be flexible in various languages and domains. The code, trained model and synthetic data are available at https://github.com/clovaai/donut.

Motivation & Objective

• Address the high computational cost, inflexibility, and error propagation issues inherent in OCR-dependent VDU pipelines.
• Develop an end-to-end document understanding model that bypasses OCR by directly learning to map raw images to structured outputs.
• Enable multi-lingual and domain-flexible document understanding through synthetic data generation and pre-training.
• Demonstrate that a simple Transformer-based architecture with minimal pre-training objective can outperform complex OCR-based pipelines in speed and accuracy.

Proposed method

• Propose Donut, a vision-and-language Transformer model that takes raw document images as input and generates structured outputs (e.g., JSON) directly.
• Use a pre-training objective based on next-token prediction, conditioning on both image features and previous text tokens to learn text reading without OCR.
• Leverage a synthetic document image generator, SynthDoG, to create diverse, multi-lingual training data for pre-training.
• Adopt a pre-train-and-fine-tune paradigm: pre-train on synthetic data for text understanding, then fine-tune on downstream VDU tasks.
• Train the model end-to-end using cross-entropy loss between predicted and ground-truth text sequences.
• Enable domain and language generalization by pre-training on synthetic data across multiple languages and document types.

Experimental results

Research questions

• RQ1 Can a simple, end-to-end Transformer-based model achieve state-of-the-art performance in VDU without relying on OCR?
• RQ2 How does the performance of an OCR-free model compare to OCR-based baselines in terms of speed, accuracy, and robustness under low-resource settings?
• RQ3 To what extent can synthetic data generated by SynthDoG enable multi-lingual and domain-general VDU without real-world labeled data?
• RQ4 Does pre-training on a simple text prediction objective enable strong holistic document understanding capabilities?

Key findings

• Donut achieves state-of-the-art performance on multiple VDU benchmarks, including CORD, FUNSD, and SROIE, outperforming OCR-based models in both accuracy and inference speed.
• On the CORD dataset, Donut achieves 95.4% F1-score on document parsing with only 10% of the training data (80 samples), outperforming LayoutLMv2 under low-resource conditions.
• The model demonstrates robustness to low-resolution inputs, maintaining strong performance even at 2560×1920 resolution with minimal training data.
• SynthDoG enables effective pre-training on synthetic data, allowing Donut to generalize across languages and domains without retraining OCR components.
• Donut reduces computational cost by eliminating the need for separate OCR inference, achieving faster end-to-end inference than OCR-based pipelines.
• The ablation study confirms that the pre-training objective focused on text prediction is sufficient to learn both text reading and holistic document understanding.