[Paper Review] A human-inspired recognition system for premodern Japanese historical documents

Anh Duc Le, Tarin Clanuwat+1 more|arXiv (Cornell University)|May 14, 2019

Handwritten Text Recognition Techniques14 references4 citations

TL;DR

This paper proposes a human-inspired attention-based encoder-decoder system for recognizing premodern Japanese historical documents, mimicking human reading by detecting line starts, scanning characters sequentially, and handling cursive, connected script. It achieves a 9.87% sequence error rate on level 2 and 53.81% on level 3 of the PRMU Kuzushiji competition dataset, outperforming all competing systems.

ABSTRACT

Recognition of historical documents is a challenging problem due to the noised, damaged characters and background. However, in Japanese historical documents, not only contains the mentioned problems, pre-modern Japanese characters were written in cursive and are connected. Therefore, character segmentation based methods do not work well. This leads to the idea of creating a new recognition system. In this paper, we propose a human-inspired document reading system to recognize multiple lines of premodern Japanese historical documents. During the reading, people employ eyes movement to determine the start of a text line. Then, they move the eyes from the current character/word to the next character/word. They can also determine the end of a line or skip a figure to move to the next line. The eyes movement integrates with visual processing to operate the reading process in the brain. We employ attention-based encoder-decoder to implement this recognition system. First, the recognition system detects where to start a text line. Second, the system scans and recognize character by character until the text line is completed. Then, the system continues to detect the start of the next text line. This process is repeated until reading the whole document. We tested our human-inspired recognition system on the pre-modern Japanese historical document provide by the PRMU Kuzushiji competition. The results of the experiments demonstrate the superiority and effectiveness of our proposed system by achieving Sequence Error Rate of 9.87% and 53.81% on level 2 and level 3 of the dataset, respectively. These results outperform to any other systems participated in the PRMU Kuzushiji competition.

Motivation & Objective

• To address the challenge of recognizing cursive, connected premodern Japanese characters in damaged historical documents where traditional segmentation-based methods fail.
• To model human reading behavior—eye movement and sequential scanning—into a deep learning system for improved recognition of complex scripts.
• To develop a sequence-to-sequence recognition system that detects line starts, processes text line-by-line, and handles variable-length lines without prior segmentation.
• To outperform existing systems in the PRMU Kuzushiji competition by leveraging human-like visual attention and sequential processing.

Proposed method

• The system uses an attention-based encoder-decoder architecture to process document images in a sequence-to-sequence manner.
• It first detects the starting position of each text line using spatial attention mechanisms.
• The model then performs character-by-character recognition through sequential scanning, mimicking human eye movement.
• Attention mechanisms allow the decoder to focus on relevant image regions at each step, improving context awareness.
• The system processes one line at a time, detecting line endings and moving to the next line until the entire document is read.
• The architecture is trained end-to-end on the Kuzushiji dataset using sequence-level loss to minimize error rate.

Experimental results

Research questions

• RQ1 Can a recognition system inspired by human visual attention and eye movement patterns improve performance on cursive, connected premodern Japanese scripts?
• RQ2 How does sequential, line-by-line processing compare to end-to-end or segmentation-based approaches for historical Japanese documents?
• RQ3 To what extent can attention mechanisms enhance recognition accuracy in the presence of noise, damage, and script variation?
• RQ4 Does modeling human reading behavior lead to better generalization and lower error rates on challenging historical document benchmarks?

Key findings

• The proposed system achieved a sequence error rate (SER) of 9.87% on level 2 of the Kuzushiji competition dataset, outperforming all other systems.
• On level 3, the system achieved an SER of 53.81%, which was the best result among all participants in the competition.
• The human-inspired attention mechanism enabled robust recognition of cursive and connected characters without relying on character segmentation.
• The system's performance demonstrates the effectiveness of modeling human-like sequential scanning and line detection in document recognition.
• The results confirm that attention-based sequence modeling is highly effective for complex historical scripts with high visual variability.
• The system's architecture generalizes well across different document qualities and script styles present in premodern Japanese manuscripts.