[Paper Review] Single Shot TextSpotter with Explicit Alignment and Attention

Tong He, Zhi Tian+4 more|arXiv (Cornell University)|Mar 9, 2018

Handwritten Text Recognition Techniques3 citations

TL;DR

This paper proposes a single-shot text spotting framework that jointly detects and recognizes text in natural images using an explicit text-alignment layer and character-level attention. By sharing features across detection and recognition tasks through an end-to-end trainable model, it achieves state-of-the-art performance on ICDAR2015, improving F-measure from 0.54 to 0.82 with a strong lexicon.

ABSTRACT

Text detection and recognition in natural images have long been considered as two separate tasks that are processed sequentially. Training of two tasks in a unified framework is non-trivial due to significant dif- ferences in optimisation difficulties. In this work, we present a conceptually simple yet efficient framework that simultaneously processes the two tasks in one shot. Our main contributions are three-fold: 1) we propose a novel text-alignment layer that allows it to precisely compute convolutional features of a text instance in ar- bitrary orientation, which is the key to boost the per- formance; 2) a character attention mechanism is introduced by using character spatial information as explicit supervision, leading to large improvements in recognition; 3) two technologies, together with a new RNN branch for word recognition, are integrated seamlessly into a single model which is end-to-end trainable. This allows the two tasks to work collaboratively by shar- ing convolutional features, which is critical to identify challenging text instances. Our model achieves impressive results in end-to-end recognition on the ICDAR2015 dataset, significantly advancing most recent results, with improvements of F-measure from (0.54, 0.51, 0.47) to (0.82, 0.77, 0.63), by using a strong, weak and generic lexicon respectively. Thanks to joint training, our method can also serve as a good detec- tor by achieving a new state-of-the-art detection performance on two datasets.

Motivation & Objective

• To unify text detection and recognition into a single, end-to-end trainable framework.
• To address the challenge of optimizing two fundamentally different tasks simultaneously.
• To improve recognition accuracy by explicitly supervising character spatial positions via attention.
• To enhance detection performance through collaborative learning between detection and recognition branches.
• To enable precise feature extraction for arbitrarily oriented text using a novel alignment layer.

Proposed method

• A novel text-alignment layer is introduced to compute convolutional features of text instances at arbitrary orientations.
• Character spatial information is used as explicit supervision in a character attention mechanism to improve recognition.
• A new RNN branch is integrated into the model for word-level recognition.
• The detection and recognition branches share shared convolutional features, enabling collaborative learning.
• The entire model is trained end-to-end, allowing joint optimization of detection and recognition.
• The framework seamlessly integrates alignment, attention, and RNN-based recognition into a single inference pipeline.

Experimental results

Research questions

• RQ1 Can joint training of text detection and recognition improve performance on both tasks?
• RQ2 How does explicit supervision via character spatial attention affect recognition accuracy?
• RQ3 To what extent does the text-alignment layer improve performance on arbitrarily oriented text?
• RQ4 Can a unified single-shot model outperform sequential or two-stage approaches?
• RQ5 Does shared feature learning between detection and recognition lead to better generalization on challenging text instances?

Key findings

• The model achieves an F-measure of 0.82 on ICDAR2015 with a strong lexicon, a significant improvement from the previous state-of-the-art of 0.54.
• With a weak lexicon, the F-measure reaches 0.77, up from 0.51 in prior work.
• For a generic lexicon, the F-measure improves from 0.47 to 0.63, demonstrating robustness across lexicon types.
• The model sets a new state-of-the-art detection performance on two benchmark datasets due to joint training.
• The integration of explicit alignment and character attention leads to substantial gains in recognition accuracy, especially for challenging text instances.