[Paper Review] STN-OCR: A single Neural Network for Text Detection and Text Recognition

Christian Bartz, Haojin Yang+1 more|arXiv (Cornell University)|Jul 27, 2017

Handwritten Text Recognition TechniquesComputer Science23 references50 citations

TL;DR

STN-OCR presents a single end-to-end deep neural network that jointly detects and recognizes text in natural scenes using a spatial transformer as an attention-based localization module, trained semi-supervisedly.

ABSTRACT

Detecting and recognizing text in natural scene images is a challenging, yet not completely solved task. In re- cent years several new systems that try to solve at least one of the two sub-tasks (text detection and text recognition) have been proposed. In this paper we present STN-OCR, a step towards semi-supervised neural networks for scene text recognition, that can be optimized end-to-end. In contrast to most existing works that consist of multiple deep neural networks and several pre-processing steps we propose to use a single deep neural network that learns to detect and recognize text from natural images in a semi-supervised way. STN-OCR is a network that integrates and jointly learns a spatial transformer network, that can learn to detect text regions in an image, and a text recognition network that takes the identified text regions and recognizes their textual content. We investigate how our model behaves on a range of different tasks (detection and recognition of characters, and lines of text). Experimental results on public benchmark datasets show the ability of our model to handle a variety of different tasks, without substantial changes in its overall network structure.

Motivation & Objective

• Motivate end-to-end scene text recognition without separate detection/recognition pipelines.
• Propose a single DNN that jointly learns text localization and recognition using a recurrent spatial transformer.
• Show semi-supervised training where text localization is learned via backpropagation from recognition losses.
• Demonstrate competitive or state-of-the-art performance on standard scene text benchmarks across detection and recognition tasks.
• Provide practical training guidance and release code/models for the research community.

Proposed method

• Integrates a localization network that predicts affine transformations for sampling text regions via a recurrent spatial transformer.
• Uses a CNN-based recognition network (ResNet variant) to recognize text from N extracted crops.
• Applies bilinear sampling for differentiable grid-based region extraction enabling end-to-end backpropagation.
• Employs either fixed-length softmax classifiers per position or CTC for sequence prediction in recognition.
• Trains the model end-to-end with text labels only for the image-level content, without ground-truth for text localization.
• Advises a two-stage pre-training strategy (SGD for pre-training, followed by Adam for finetuning) to improve convergence on more challenging tasks.

Experimental results

Research questions

• RQ1 Can a single multi-task neural network learn to both detect text regions and recognize text in natural scenes in an end-to-end fashion?
• RQ2 Does integrating a spatial transformer as a learned attention mechanism improve end-to-end training and performance for scene text detection and recognition?
• RQ3 What are effective training strategies to converge such a model when detecting multiple lines of text?
• RQ4 How does semi-supervised localization (driven by recognition loss) compare to fully supervised or hand-engineered pipelines on standard benchmarks?

Key findings

• The proposed STN-OCR network achieves competitive to state-of-the-art performance on several standard scene text benchmarks in end-to-end fashion.
• The system can learn text localization in a semi-supervised way by backpropagating recognition losses through the spatial transformer, without explicit localization labels.
• A two-stage training protocol (start with SGD on simpler tasks, then switch to Adam for fine-tuning) improves convergence and localization quality.
• Using a ResNet-based backbone in both the localization and recognition stages yields better gradient flow and recognition performance.
• On robust reading datasets (ICDAR 2013, SVT, IIIT5K) the method achieves competitive recognition accuracies without external lexicons beyond standard post-processing.
• Preliminary experiments on the FSNS dataset indicate the model can localize and recognize individual words, though full multi-line text detection remains challenging.