[Paper Review] 2D Attentional Irregular Scene Text Recognizer

Pengyuan Lyu, Zhicheng Yang+4 more|arXiv (Cornell University)|Jun 13, 2019

Handwritten Text Recognition Techniques46 references50 citations

TL;DR

The paper presents a 2D attention-based framework that directly encodes and decodes irregular scene text in 2D space using a relation attention module and a parallel attention module, achieving state-of-the-art results and faster inference on multiple benchmarks.

ABSTRACT

Irregular scene text, which has complex layout in 2D space, is challenging to most previous scene text recognizers. Recently, some irregular scene text recognizers either rectify the irregular text to regular text image with approximate 1D layout or transform the 2D image feature map to 1D feature sequence. Though these methods have achieved good performance, the robustness and accuracy are still limited due to the loss of spatial information in the process of 2D to 1D transformation. Different from all of previous, we in this paper propose a framework which transforms the irregular text with 2D layout to character sequence directly via 2D attentional scheme. We utilize a relation attention module to capture the dependencies of feature maps and a parallel attention module to decode all characters in parallel, which make our method more effective and efficient. Extensive experiments on several public benchmarks as well as our collected multi-line text dataset show that our approach is effective to recognize regular and irregular scene text and outperforms previous methods both in accuracy and speed.

Motivation & Objective

• Motivate robust recognition of irregular, 2D-layout text without rectification or 1D conversion.
• Develop an end-to-end framework that preserves 2D spatial information for improved accuracy.
• Introduce a 2D relation attention module to capture global context.
• Introduce a parallel attention module to output multiple characters simultaneously.
• Demonstrate state-of-the-art performance and efficiency on regular, irregular, and multi-line text datasets.

Proposed method

• Encode input with a CNN encoder to obtain 2D feature maps.
• Apply a 2D relation attention module (multi-layer bidirectional transformer) to capture global dependencies across flattened feature maps.
• Use a parallel attention module to produce multiple glimpses from the 2D features in parallel (output nodes n).
• Decode glimpses with a two-stage decoder: first-stage decoder predicts preliminary characters; second-stage decoder, with a relation attention module, models dependencies among outputs and refines predictions.
• Train end-to-end with a multi-task loss that sums cross-entropy over two decoders and all output nodes.

Experimental results

Research questions

• RQ1 Can 2D attention directly operate on irregular 2D text layouts to produce accurate character sequences without rectification or 1D flattening?
• RQ2 Do relation attention and parallel attention improve robustness and efficiency for irregular and multi-line text recognition?
• RQ3 How does the proposed method perform on regular and irregular datasets and on multi-line text like license plates?
• RQ4 What is the impact of the second-stage decoder and the number of transformer layers on recognition accuracy?

Key findings

• Achieves state-of-the-art results on multiple regular and irregular scene text datasets.
• Outperforms previous methods on SVTP and CUTE80 by 3.8% and 3.5%, respectively.
• Demonstrates strong multi-line text recognition on MLT280, outperforming ASTER and SAR by large margins (61.4% vs 40.0% random init and 80.7% vs 62.5% fine-tuned).
• Is faster than rectification-based and recurrent 2D attention methods, by about 2.1x and 4.4x respectively on MLT280.
• Second-stage decoder and relation attention module consistently improve accuracy over the first-stage decoder and baseline variants.