[Paper Review] DocAligner: Annotating Real-world Photographic Document Images by Simply Taking Pictures

Jiaxin Zhang, Bangdong Chen+4 more|arXiv (Cornell University)|Jun 9, 2023

Handwritten Text Recognition Techniques4 citations

TL;DR

DocAligner automates annotation of real-world photographic document images by establishing dense correspondence between photographic and clean document images, enabling automatic transfer of existing annotations and generation of new labels through non-rigid alignment, hierarchical correlation, and recurrent refinement. It reduces manual labeling time by at least 30-fold and achieves state-of-the-art performance on layout analysis, illumination correction, and geometric rectification tasks.

ABSTRACT

Recently, there has been a growing interest in research concerning document image analysis and recognition in photographic scenarios. However, the lack of labeled datasets for this emerging challenge poses a significant obstacle, as manual annotation can be time-consuming and impractical. To tackle this issue, we present DocAligner, a novel method that streamlines the manual annotation process to a simple step of taking pictures. DocAligner achieves this by establishing dense correspondence between photographic document images and their clean counterparts. It enables the automatic transfer of existing annotations in clean document images to photographic ones and helps to automatically acquire labels that are unavailable through manual labeling. Considering the distinctive characteristics of document images, DocAligner incorporates several innovative features. First, we propose a non-rigid pre-alignment technique based on the document's edges, which effectively eliminates interference caused by significant global shifts and repetitive patterns present in document images. Second, to handle large shifts and ensure high accuracy, we introduce a hierarchical aligning approach that combines global and local correlation layers. Furthermore, considering the importance of fine-grained elements in document images, we present a details recurrent refinement module to enhance the output in a high-resolution space. To train DocAligner, we construct a synthetic dataset and introduce a self-supervised learning approach to enhance its robustness for real-world data. Through extensive experiments, we demonstrate the effectiveness of DocAligner and the acquired dataset. Datasets and codes will be publicly available.

Motivation & Objective

• To address the critical lack of labeled photographic document image datasets, which hinders progress in document AI for real-world scenarios.
• To eliminate the need for costly and time-consuming manual annotation by enabling automatic label generation through image correspondence.
• To develop a robust method for dense correspondence between photographic and clean document images, despite non-rigid deformations, perspective shifts, and repetitive patterns.
• To construct a high-quality synthetic dataset and a self-supervised training strategy to improve generalization to real-world data.
• To demonstrate the practical utility of the method across multiple document AI tasks, including layout analysis, illumination correction, and geometric rectification.

Proposed method

• Proposes a non-rigid pre-alignment using thin-plate splines (TPS) to handle global shifts and repetitive patterns in document images, improving initial alignment robustness.
• Introduces a hierarchical alignment module combining global and local correlation layers with coarse-to-fine flow prediction to achieve high-accuracy dense correspondence.
• Designs a details recurrent refinement module using memory-efficient ConvGRU to enhance high-resolution flow outputs, preserving fine-grained character-level details.
• Constructs a synthetic dataset using PDF-based clean images warped with random flows and degraded with real shading maps to simulate photographic distortions.
• Employs a self-supervised learning approach during training to improve generalization on real-world photographic data without requiring ground-truth flow annotations.
• Utilizes a Transformer-based architecture for downstream tasks like geometric rectification, trained on real data acquired via DocAligner, achieving SOTA performance.

Figure 1 : Automatic annotation for real-world photographic document images via DocAligner. All you need to do is take pictures.

Experimental results

Research questions

• RQ1 Can dense correspondence between clean and photographic document images be established effectively despite non-rigid deformations and complex patterns?
• RQ2 Can automatic annotation be reduced to simply taking pictures of printed documents, eliminating manual labeling?
• RQ3 Does the proposed non-rigid pre-alignment outperform rigid transformations like homography in handling diverse document distortions?
• RQ4 Can the hierarchical alignment and recurrent refinement modules significantly improve flow accuracy and detail preservation in high-resolution document images?
• RQ5 Can a self-supervised training strategy on synthetic data combined with real data acquisition yield superior performance on real-world document AI tasks?

Key findings

• DocAligner reduces manual labeling time by a minimum of 30-fold when annotating layout analysis data, significantly lowering annotation costs.
• The model achieves a MS-SSIM of 0.8058 and AD of 0.0486 on the DocUNet benchmark when using both non-rigid pre-alignment and details recurrent refinement, outperforming ablated variants.
• A Transformer-based geometric rectification model trained on real data acquired via DocAligner achieves a MS-SSIM of 0.4897 and AD of 0.4226, surpassing models trained on larger synthetic datasets.
• Self-supervised training on the synthetic dataset improves generalization, enabling strong performance on real data even with limited real annotations.
• The ablation study confirms that non-rigid pre-alignment and details recurrent refinement are both essential, with performance dropping significantly when either is removed.
• The method enables automatic label transfer from clean document images to photographic ones, enabling rapid data creation for tasks like layout analysis and table recognition.

Figure 2 : Overall architecture of DocAligner. Considering photographic $I_{s}$ and clean $I_{t}$ , DocAligner aims to densely correlate each pixel in $I_{t}$ with $I_{s}$ .