[Paper Review] Learning to Extract Semantic Structure from Documents Using Multimodal Fully Convolutional Neural Network

Xiao Yang, Ersin Yumer+4 more|arXiv (Cornell University)|Jun 7, 2017

Handwritten Text Recognition Techniques58 citations

TL;DR

This paper presents a multimodal fully convolutional network (MFCN) that jointly uses visual features and text embeddings to perform pixel-wise semantic structure extraction from document images, aided by synthetic data pretraining and unsupervised training tasks.

ABSTRACT

We present an end-to-end, multimodal, fully convolutional network for extracting semantic structures from document images. We consider document semantic structure extraction as a pixel-wise segmentation task, and propose a unified model that classifies pixels based not only on their visual appearance, as in the traditional page segmentation task, but also on the content of underlying text. Moreover, we propose an efficient synthetic document generation process that we use to generate pretraining data for our network. Once the network is trained on a large set of synthetic documents, we fine-tune the network on unlabeled real documents using a semi-supervised approach. We systematically study the optimum network architecture and show that both our multimodal approach and the synthetic data pretraining significantly boost the performance.

Motivation & Objective

• Develop an end-to-end model that simultaneously identifies appearance-based (layout) and semantics-based (document roles) classes at the pixel level.
• Incorporate textual information via a text embedding map to improve segmentation, especially for small or semantically distinctive regions.
• Leverage synthetic data to scale supervision and propose unsupervised auxiliary tasks to enhance representation learning.
• Enable semi-supervised training by alternating between synthetic labeled data and unlabeled real documents.

Proposed method

• Propose a multimodal fully convolutional network with an encoder, a segmentation decoder, an auxiliary reconstruction decoder (training only), and a bridge that fuses visual and textual representations.
• Construct a text embedding map by averaging word embeddings to create sentence-level vectors, assign them to corresponding sentence regions, and concatenate with visual features.
• Introduce a dilated convolution block to expand receptive fields for small objects while preserving spatial accuracy via unpooling-based skip connections.
• Train text embeddings using a skip-gram model on Wikipedia to obtain 128-dimensional word vectors for constructing the embedding map.
• Use two unsupervised losses during training: a reconstruction loss (via an auxiliary decoder) and a consistency loss (enforcing intra-region feature similarity using bounding boxes).
• Develop a large-scale synthetic document generator (135,000 pages) to provide pixel-wise ground truth for supervised pretraining, then fine-tune on real documents in a semi-supervised manner.

Experimental results

Research questions

• RQ1 Can a unified multimodal network outperform appearance-only approaches in document segmentation by leveraging underlying text content?
• RQ2 What is the impact of incorporating a text embedding map on pixel-wise semantic segmentation of documents?
• RQ3 Do unsupervised auxiliary tasks (reconstruction and consistency) improve representation learning and segmentation accuracy on real documents?
• RQ4 How effective is synthetic data pretraining for enabling high-performance document semantic structure extraction on real data?

Key findings

• The MFCN with multimodal fusion significantly improves segmentation performance over appearance-based baselines on DSSE-200, ICDAR2015, and SectLabel datasets.
• Incorporating a text embedding map improves accuracy for semantic classes such as section headings, captions, lists, and paragraphs.
• Unsupervised tasks provide gains, with consistency loss contributing a notable improvement when combined with reconstruction loss.
• Using real (ground-truth) text in the embedding map yields substantial performance gains over using OCR-extracted text or no text information, indicating the value of accurate textual context.
• On ICDAR2015, the model achieves IoU scores of 94.5 for non-text, 91.0 for text, and 77.1 for figure regions in a binary variant, outperforming prior methods in these categories.
• On SectLabel, the model attains competitive F1 scores for section headings (0.919), captions (0.893), and lists (0.793), while also identifying figures and tables.