[Paper Review] Spatial Dual-Modality Graph Reasoning for Key Information Extraction

Hongbin Sun, Zhanghui Kuang+3 more|arXiv (Cornell University)|Mar 26, 2021

Handwritten Text Recognition Techniques36 references16 citations

TL;DR

This paper proposes Spatial Dual-Modality Graph Reasoning (SDMG-R), an end-to-end method for key information extraction from unstructured document images that models text regions as dual-modality graphs combining visual and textual features with spatial relations. SDMG-R uses iterative graph convolutional message passing to refine node representations, achieving state-of-the-art performance on SROIE and a new, larger, and more realistic benchmark, WildReceipt, with a 4.4% absolute F1 improvement when including key category classification.

ABSTRACT

Key information extraction from document images is of paramount importance in office automation. Conventional template matching based approaches fail to generalize well to document images of unseen templates, and are not robust against text recognition errors. In this paper, we propose an end-to-end Spatial Dual-Modality Graph Reasoning method (SDMG-R) to extract key information from unstructured document images. We model document images as dual-modality graphs, nodes of which encode both the visual and textual features of detected text regions, and edges of which represent the spatial relations between neighboring text regions. The key information extraction is solved by iteratively propagating messages along graph edges and reasoning the categories of graph nodes. In order to roundly evaluate our proposed method as well as boost the future research, we release a new dataset named WildReceipt, which is collected and annotated tailored for the evaluation of key information extraction from document images of unseen templates in the wild. It contains 25 key information categories, a total of about 69000 text boxes, and is about 2 times larger than the existing public datasets. Extensive experiments validate that all information including visual features, textual features and spatial relations can benefit key information extraction. It has been shown that SDMG-R can effectively extract key information from document images of unseen templates, and obtain new state-of-the-art results on the recent popular benchmark SROIE and our WildReceipt. Our code and dataset will be publicly released.

Motivation & Objective

• Address the limitation of template-based methods that fail to generalize to unseen document layouts and are sensitive to text recognition errors.
• Develop a robust, end-to-end method for key information extraction that leverages both visual and textual features and spatial context beyond line-level alignment.
• Create a new, large-scale benchmark dataset, WildReceipt, to evaluate generalization to unseen templates in realistic, unconstrained image conditions.
• Demonstrate that joint reasoning over dual-modality features and spatial graph structures significantly improves extraction accuracy.

Proposed method

• Model document images as spatial dual-modality graphs where nodes represent detected text boxes with visual (CNN) and textual (RNN) features.
• Define graph edges based on 2D spatial proximity between neighboring text regions to encode relational context.
• Use a graph convolutional network (GCN) with learnable attention weights to iteratively propagate and refine node representations across edges.
• Introduce a Kronecker-product-based block diagonal tensor decomposition for effective fusion of visual and textual features into a joint representation.
• Apply multi-head attention in the graph reasoning module to dynamically weight the importance of neighboring nodes during message passing.
• Train the model end-to-end using cross-entropy loss for both key category and value category classification.

Experimental results

Research questions

• RQ1 Can joint reasoning over visual, textual, and spatial relational features improve key information extraction beyond line-level context?
• RQ2 How does the inclusion of spatial relations between non-collinear text regions affect performance on unseen templates?
• RQ3 To what extent do dual-modality features (visual + textual) contribute to robustness against recognition errors and layout variations?
• RQ4 How does the number of graph reasoning iterations influence model performance and generalization?
• RQ5 Can a new, fine-grained, and realistic dataset like WildReceipt better evaluate generalization than existing benchmarks?

Key findings

• SDMG-R achieves a new state-of-the-art F1 score of 88.7% on the WildReceipt dataset, significantly outperforming prior methods.
• Removing graph reasoning leads to an 11.5% absolute drop in F1 score on WildReceipt, proving that spatial message passing is critical for performance.
• Including key category classification improves value category classification by 4.4% absolute F1, showing its auxiliary benefit.
• The optimal number of graph reasoning iterations is L=2, with performance degrading at L>2 due to overfitting.
• The proposed dual modality fusion module outperforms alternatives like LinearSum and ConcatMLP, especially with optimal block size and number of blocks.
• Visualization confirms that the model learns meaningful, semantically relevant spatial attention weights, even across long distances, such as linking "Total value" with "Total key".