[Paper Review] Candidate Fusion: Integrating Language Modelling into a Sequence-to-Sequence Handwritten Word Recognition Architecture

Lei Kang, Pau Riba+3 more|arXiv (Cornell University)|Dec 21, 2019

Handwritten Text Recognition Techniques47 references4 citations

TL;DR

This paper proposes Candidate Fusion, a novel method to integrate an external language model into a sequence-to-sequence handwritten word recognition system by jointly training the recognizer and language model. The approach enables dynamic fusion of language model predictions with the recognizer’s output, adaptively learning from recognition errors and outperforming state-of-the-art methods on public datasets and real industrial use cases.

ABSTRACT

Sequence-to-sequence models have recently become very popular for tackling handwritten word recognition problems. However, how to effectively integrate an external language model into such recognizer is still a challenging problem. The main challenge faced when training a language model is to deal with the language model corpus which is usually different to the one used for training the handwritten word recognition system. Thus, the bias between both word corpora leads to incorrectness on the transcriptions, providing similar or even worse performances on the recognition task. In this work, we introduce Candidate Fusion, a novel way to integrate an external language model to a sequence-to-sequence architecture. Moreover, it provides suggestions from an external language knowledge, as a new input to the sequence-to-sequence recognizer. Hence, Candidate Fusion provides two improvements. On the one hand, the sequence-to-sequence recognizer has the flexibility not only to combine the information from itself and the language model, but also to choose the importance of the information provided by the language model. On the other hand, the external language model has the ability to adapt itself to the training corpus and even learn the most commonly errors produced from the recognizer. Finally, by conducting comprehensive experiments, the Candidate Fusion proves to outperform the state-of-the-art language models for handwritten word recognition tasks.

Motivation & Objective

• Address the challenge of integrating external language models with sequence-to-sequence handwritten word recognizers trained on different corpora.
• Overcome the performance degradation caused by distributional bias between optical recognition and language model training data.
• Enable the language model to adapt to and correct common errors from the optical recognizer during joint training.
• Develop a flexible, trainable fusion mechanism that avoids manual hyperparameter tuning of the fusion weight.
• Demonstrate robustness and generalization on both public benchmarks and real-world industrial datasets.

Proposed method

• Propose Candidate Fusion, a trainable fusion mechanism that injects predictions from an external language model as an additional input token sequence to the sequence-to-sequence decoder.
• Train the sequence-to-sequence model and the external language model jointly on the same recognition data, allowing the language model to learn from and correct the recognizer’s error patterns.
• Use attention mechanisms in the decoder to dynamically attend to both the visual features and the candidate transcriptions from the language model.
• Integrate synthetic handwriting data and data augmentation strategies during pre-training to improve generalization and robustness.
• Apply a learnable fusion gate or attention-based weighting to control the influence of the language model predictions during decoding.
• Use edit-distance-based lexicon constraints in ablation studies to evaluate performance under vocabulary restrictions.

Figure 10: A cropped area of the real industrial use case dataset.

Experimental results

Research questions

• RQ1 Can joint training of a sequence-to-sequence recognizer and an external language model reduce performance degradation due to corpus mismatch?
• RQ2 How does Candidate Fusion compare to shallow and deep fusion baselines in terms of recognition accuracy and robustness?
• RQ3 To what extent can the language model adapt to and correct common errors produced by the optical recognizer during training?
• RQ4 Does the proposed method generalize to real-world industrial datasets with complex, noisy handwritten forms?
• RQ5 Can Candidate Fusion outperform state-of-the-art approaches without relying on a closed lexicon?

Key findings

• Candidate Fusion reduces CER from 5.79 to 5.74 on the IAM dataset and from 2.82 to 2.52 on GW, outperforming both shallow and deep fusion baselines.
• On the Rimes dataset, Candidate Fusion achieves a CER of 2.32 and WER of 7.47, significantly outperforming the baseline (CER 2.65, WER 8.71) and other fusion methods.
• With a closed lexicon, Candidate Fusion achieves a CER of 1.45 on Rimes (using only test set lexicon), reducing error rates by over 40% compared to the baseline.
• In a real industrial use case involving car crash statement forms, Candidate Fusion achieved a CER of 3.7 and WER of 4.5, outperforming a strong CTC-based baseline (CER 5.6, WER 7.4).
• The ablation study confirms that joint training with the language model leads to better error correction and generalization, especially when the language model is adapted to recognizer-specific errors.