[Paper Review] End to End Recognition System for Recognizing Offline Unconstrained Vietnamese Handwriting

Anh Duc Le, Hung Tuan Nguyen+1 more|arXiv (Cornell University)|May 14, 2019

Handwritten Text Recognition Techniques22 references4 citations

TL;DR

This paper proposes an end-to-end attention-based encoder-decoder model (AED) for offline unconstrained Vietnamese handwriting recognition, combining DenseNet for feature extraction and an LSTM decoder with attention for sequence generation. The model achieves a 12.30% word error rate on the VNOnDB-Word dataset without external language models, outperforming baseline systems and matching Google's performance in a public competition.

ABSTRACT

Inspired by recent successes in neural machine translation and image caption generation, we present an attention based encoder decoder model (AED) to recognize Vietnamese Handwritten Text. The model composes of two parts: a DenseNet for extracting invariant features, and a Long Short-Term Memory network (LSTM) with an attention model incorporated for generating output text (LSTM decoder), which are connected from the CNN part to the attention model. The input of the CNN part is a handwritten text image and the target of the LSTM decoder is the corresponding text of the input image. Our model is trained end-to-end to predict the text from a given input image since all the parts are differential components. In the experiment section, we evaluate our proposed AED model on the VNOnDB-Word and VNOnDB-Line datasets to verify its efficiency. The experiential results show that our model achieves 12.30% of word error rate without using any language model. This result is competitive with the handwriting recognition system provided by Google in the Vietnamese Online Handwritten Text Recognition competition.

Motivation & Objective

• To develop an end-to-end deep learning system for recognizing unconstrained Vietnamese handwritten text without relying on handcrafted features or external language models.
• To improve recognition accuracy for Vietnamese, a language with complex diacritics and tonal marks, using a unified neural architecture.
• To evaluate the model’s performance on standard benchmarks: VNOnDB-Word and VNOnDB-Line datasets.
• To demonstrate competitive performance against state-of-the-art systems, including Google’s solution, in a public competition setting.

Proposed method

• The model uses a DenseNet backbone to extract hierarchical, spatially invariant features from handwritten image inputs.
• An attention mechanism is integrated into an LSTM decoder to dynamically focus on relevant image regions during text generation.
• The entire system is trained end-to-end using backpropagation, with all components being differentiable.
• The encoder processes the input image to generate a context-aware feature representation.
• The decoder generates text tokens autoregressively, conditioned on the encoded features and attention over the image.
• The loss function is based on cross-entropy between predicted and ground-truth text sequences.

Experimental results

Research questions

• RQ1 Can an end-to-end attention-based encoder-decoder model achieve competitive performance on unconstrained Vietnamese handwriting recognition without language models?
• RQ2 How effective is the combination of DenseNet and attention-enhanced LSTM for capturing complex visual and sequential patterns in Vietnamese handwriting?
• RQ3 Does the proposed model outperform existing systems on standard benchmarks like VNOnDB-Word and VNOnDB-Line?
• RQ4 To what extent does the model generalize to unconstrained writing styles common in real-world applications?

Key findings

• The proposed AED model achieves a word error rate (WER) of 12.30% on the VNOnDB-Word dataset without using any external language model.
• The model's performance is competitive with Google's system, which was the top performer in the Vietnamese Online Handwritten Text Recognition competition.
• The integration of attention with LSTM improves alignment between image features and generated text, enhancing recognition accuracy.
• The end-to-end training strategy enables joint optimization of feature extraction and sequence generation, reducing error propagation.