[Paper Review] Improving Long Handwritten Text Line Recognition with Convolutional Multi-way Associative Memory

Duc Dung Nguyen, Nhan Viet Tran+1 more|arXiv (Cornell University)|Nov 5, 2019

Handwritten Text Recognition Techniques42 references4 citations

TL;DR

This paper proposes Convolutional Multi-way Associative Memory (CMAM), a memory-augmented CRNN architecture that enhances long handwritten text line recognition by integrating an external memory module with multi-way associative access and refinement steps. CMAM significantly reduces character error rates—by up to 14% on long, complex Japanese text lines—outperforming standard CRNNs and DNCs, especially in long-context scenarios.

ABSTRACT

Convolutional Recurrent Neural Networks (CRNNs) excel at scene text recognition. Unfortunately, they are likely to suffer from vanishing/exploding gradient problems when processing long text images, which are commonly found in scanned documents. This poses a major challenge to goal of completely solving Optical Character Recognition (OCR) problem. Inspired by recently proposed memory-augmented neural networks (MANNs) for long-term sequential modeling, we present a new architecture dubbed Convolutional Multi-way Associative Memory (CMAM) to tackle the limitation of current CRNNs. By leveraging recent memory accessing mechanisms in MANNs, our architecture demonstrates superior performance against other CRNN counterparts in three real-world long text OCR datasets.

Motivation & Objective

• Address the vanishing/exploding gradient problem in CRNNs when processing long handwritten text lines.
• Overcome the limitation of RNN-based decoders in capturing long-range contextual dependencies in handwritten text.
• Enhance sequence modeling in HTR by integrating an external memory module inspired by memory-augmented neural networks (MANNs).
• Demonstrate that memory-augmented architectures can outperform standard CRNNs on long, complex text recognition tasks.
• Evaluate the proposed CMAM model on diverse datasets to validate its robustness and generalization across languages and text lengths.

Proposed method

• Integrate an external memory module into a convolutional recurrent neural network (CRNN) to replace standard RNN-based decoders.
• Use a controller network with two heads: one for writing features into memory and one for reading from memory via differentiable attention.
• Implement multi-way associative memory access, allowing multiple memory slots to be accessed simultaneously for improved context modeling.
• Apply iterative refinement steps (l=1) to enhance memory read accuracy and feature representation over multiple timesteps.
• Train the end-to-end model using Connectionist Temporal Classification (CTC) loss for sequence-to-sequence alignment.
• Pre-train on synthetic Japanese text images (100,000 samples) before fine-tuning on real-world scanned documents to improve data efficiency.

Experimental results

Research questions

• RQ1 Can a memory-augmented architecture improve long-context handwritten text recognition beyond standard CRNNs?
• RQ2 How does multi-way associative memory access enhance sequence modeling in handwritten text recognition compared to standard RNNs?
• RQ3 Does the integration of external memory reduce character error rates on long and complex text lines, especially in low-resource or noisy scenarios?
• RQ4 How effective is pre-training on synthetic data for improving generalization in real-world handwritten text recognition tasks?
• RQ5 To what extent does iterative refinement in memory access improve recognition accuracy on long text lines?

Key findings

• CMAM with one refinement step (l=1) reduces character error rate (CER) by 14.29% on the validation set and 14.01% on the test set of the private Japanese dataset compared to CRNN.
• On the Japanese dataset, CMAM achieves a test CER of 10.71% after pre-training on synthetic data, outperforming CRNN’s 11.62% by 0.91 percentage points.
• On the IAM English dataset, CMAM (l=1) reduces CER by over 2% compared to standard CRNN, demonstrating consistent improvement even on shorter, cleaner lines.
• CMAM achieves a 3.04% higher correct rate (CR) than CRNN on the SCUT-EPT Chinese dataset (82.14% vs. 81.47%), confirming its superiority in complex scripts.
• The performance gap between CMAM and CRNN widens on longer, more complex text lines, indicating that memory mechanisms are particularly beneficial for long-context modeling.
• Pre-training on synthetic data significantly improves CRNN performance but yields relatively smaller gains for CMAM, suggesting CMAM is more data-efficient and robust to distribution shifts.