[Paper Review] Handwritten Text Segmentation via End-to-End Learning of Convolutional Neural Network

Junho Jo, Hyung Il Koo+2 more|arXiv (Cornell University)|Jun 12, 2019

Handwritten Text Recognition Techniques15 references4 citations

TL;DR

This paper proposes an end-to-end convolutional neural network (CNN) for handwritten text segmentation in mixed-mode documents, bypassing traditional two-step preprocessing by formulating the task as pixel-wise classification. The method uses a U-Net architecture with a novel dynamically balanced cross-entropy loss and focal loss to address class imbalance, and introduces a data synthesis pipeline to generate realistic training images with pixel-level annotations. Trained solely on synthetic data, the model improves OCR accuracy from 71.13% to 92.50% on real-world scribbled documents.

ABSTRACT

We present a new handwritten text segmentation method by training a convolutional neural network (CNN) in an end-to-end manner. Many conventional methods addressed this problem by extracting connected components and then classifying them. However, this two-step approach has limitations when handwritten components and machine-printed parts are overlapping. Unlike conventional methods, we develop an end-to-end deep CNN for this problem, which does not need any preprocessing steps. Since there is no publicly available dataset for this goal and pixel-wise annotations are time-consuming and costly, we also propose a data synthesis algorithm that generates realistic training samples. For training our network, we develop a cross-entropy based loss function that addresses the imbalance problems. Experimental results on synthetic and real images show the effectiveness of the proposed method. Specifically, the proposed network has been trained solely on synthetic images, nevertheless the removal of handwritten text in real documents improves OCR performance from 71.13% to 92.50%, showing the generalization performance of our network and synthesized images.

Motivation & Objective

• To address the challenge of separating overlapping handwritten and machine-printed text in scanned documents, which remains difficult for conventional two-step methods.
• To eliminate reliance on preprocessing steps like binarization and connected component extraction by formulating segmentation as an end-to-end pixel-wise classification task.
• To overcome the severe class imbalance between handwritten and background pixels during training through a novel loss function.
• To generate realistic synthetic training data with accurate pixel-level annotations, since no public dataset exists for this task.

Proposed method

• The method uses a U-Net-based CNN to perform pixel-wise classification, assigning '1' to handwritten text pixels and '-1' to all others (background, machine-printed text, tables, etc.).
• A new loss function, dynamically balanced cross-entropy (DBCE), is proposed to mitigate the class imbalance problem by adaptively weighting rare (handwritten) and frequent (background) classes.
• The loss function is further enhanced by incorporating focal loss to address the overwhelming problem where misclassification of machine-printed text regions degrades performance.
• A data synthesis pipeline generates realistic document images by combining scanned handwritten text (from IAM dataset), machine-printed text (from PRImA dataset), and form images, preserving texture and noise characteristics.
• The synthesis method uses image blending and geometric transformations to simulate realistic overlaps and distortions found in real documents.
• The network is trained exclusively on synthetic data, enabling generalization to real-world documents without requiring real annotated data.

Experimental results

Research questions

• RQ1 Can an end-to-end deep learning approach outperform traditional two-stage methods (e.g., connected component extraction + classification) for handwritten text segmentation in overlapping mixed-mode documents?
• RQ2 How effective is a dynamically balanced cross-entropy loss in mitigating class imbalance when training on synthetic data with sparse handwritten pixels?
• RQ3 To what extent can a model trained only on synthetic data generalize to real-world document images with complex overlaps?
• RQ4 Can a data synthesis method generate realistic training samples that preserve the visual and structural properties of real scanned documents?

Key findings

• The proposed model improves OCR accuracy from 71.13% to 92.50% on real scribbled document images after removing detected handwritten pixels, demonstrating strong generalization despite being trained only on synthetic data.
• The dynamically balanced cross-entropy loss (DBCE) significantly reduces misclassification of handwritten pixels compared to standard cross-entropy, improving IoU on synthetic data from 95.88% to 97.20%.
• The combination of DBCE and focal loss (DBCEF) achieves the highest test IoU of 97.80% on the synthetic test set, outperforming both individual components and baseline models.
• Ablation studies confirm that both DBCE and focal loss are essential: DBCE mitigates class imbalance, while focal loss reduces misclassification in machine-printed regions.
• The model achieves 99.94% IoU on the synthesized validation set when using the full DBCEF loss, indicating high segmentation accuracy on synthetic data.