[Paper Review] Neural Collaborative Graph Machines for Table Structure Recognition

Hao Liu, Xin Li+4 more|arXiv (Cornell University)|Nov 26, 2021

Handwritten Text Recognition Techniques44 references4 citations

TL;DR

This paper proposes Neural Collaborative Graph Machines (NCGM), a novel graph neural network framework that models intra- and inter-modality interactions hierarchically to improve table structure recognition (TSR) in diverse and challenging scenarios. By stacking collaborative blocks that alternate between intra-modality context extraction and inter-modality fusion, NCGM achieves state-of-the-art performance, especially on distorted and complex tables, outperforming prior methods significantly.

ABSTRACT

Recently, table structure recognition has achieved impressive progress with the help of deep graph models. Most of them exploit single visual cues of tabular elements or simply combine visual cues with other modalities via early fusion to reason their graph relationships. However, neither early fusion nor individually reasoning in terms of multiple modalities can be appropriate for all varieties of table structures with great diversity. Instead, different modalities are expected to collaborate with each other in different patterns for different table cases. In the community, the importance of intra-inter modality interactions for table structure reasoning is still unexplored. In this paper, we define it as heterogeneous table structure recognition (Hetero-TSR) problem. With the aim of filling this gap, we present a novel Neural Collaborative Graph Machines (NCGM) equipped with stacked collaborative blocks, which alternatively extracts intra-modality context and models inter-modality interactions in a hierarchical way. It can represent the intra-inter modality relationships of tabular elements more robustly, which significantly improves the recognition performance. We also show that the proposed NCGM can modulate collaborative pattern of different modalities conditioned on the context of intra-modality cues, which is vital for diversified table cases. Experimental results on benchmarks demonstrate our proposed NCGM achieves state-of-the-art performance and beats other contemporary methods by a large margin especially under challenging scenarios.

Motivation & Objective

• To address the lack of exploration in modality collaboration for heterogeneous table structures, especially under distortion and complexity.
• To propose a new problem formulation: heterogeneous table structure recognition (Hetero-TSR), emphasizing dynamic collaboration between visual, geometric, and content modalities.
• To design a deep learning framework that adaptively modulates collaboration patterns based on intra-modality context for robust performance across diverse table types.
• To overcome limitations of early and late fusion strategies, which either mix modalities too early or fail to integrate cross-modal interactions effectively.
• To enable more accurate and generalizable table structure recognition in real-world scenarios involving mobile-captured, distorted, or complex tables.

Proposed method

• The method constructs separate graphs for three modalities—appearance, geometry, and content—using text segment bounding boxes as nodes.
• It introduces a basic collaborative block composed of an Ego Context Extractor (ECE) for dynamic intra-modality context generation and a Cross Context Synthesizer (CCS) for inter-modality interaction fusion and modulation.
• The ECE uses multi-head self-attention to generate modality-specific context representations based on node features and graph topology.
• The CCS employs a learnable attention mechanism to dynamically modulate and synthesize cross-modal information, enabling context-aware collaboration.
• Multiple collaborative blocks are stacked in a hierarchical manner, allowing low-level and high-level representations to interact progressively across modalities.
• The framework is trained end-to-end with a contrastive learning objective to enhance representation quality and generalization.

Experimental results

Research questions

• RQ1 Can dynamic, context-aware collaboration between multiple modalities improve table structure recognition beyond early or late fusion?
• RQ2 How do intra- and inter-modality interactions contribute to robustness in recognizing distorted or complex tables?
• RQ3 Can the model adapt its collaboration pattern based on the intrinsic characteristics of each modality and the overall table structure?
• RQ4 What is the impact of modality diversity and interaction dynamics on recognition accuracy, especially in challenging scenarios?
• RQ5 How does the proposed hierarchical collaboration mechanism compare to standard attention or graph neural networks in modeling table relationships?

Key findings

• NCGM achieves state-of-the-art performance on benchmark datasets, outperforming existing methods such as GraphTSR, DGCNN, TabStruct-Net, and FLAG-Net.
• On the more challenging TUX and PubLayNet datasets, NCGM shows a significant margin over baselines, particularly in handling distorted and irregular table structures.
• The model reduces error rates in row and column structure prediction, especially in cases with spanning cells or misaligned boxes.
• Ablation studies confirm that both the Ego Context Extractor and Cross Context Synthesizer are essential for performance, with CCS contributing most to inter-modality collaboration gains.
• Jensen-Shannon Divergence analysis shows that CCS increases attention map diversity, indicating more effective and varied cross-modal interaction patterns.
• Qualitative results demonstrate that NCGM generalizes well to various table types, including nested and complex layouts, though it still struggles with severe misalignment in nested tables.