[Paper Review] Scene Text Recognition with Sliding Convolutional Character Models

Fei Yin, Yichao Wu+2 more|arXiv (Cornell University)|Sep 6, 2017

Handwritten Text Recognition TechniquesComputer Science32 references72 citations

TL;DR

A sliding-window CNN-based scene text recognition method that detects and recognizes characters end-to-end using convolutional character models and CTC transcription, without requiring character segmentation or lexicons.

ABSTRACT

Scene text recognition has attracted great interests from the computer vision and pattern recognition community in recent years. State-of-the-art methods use concolutional neural networks (CNNs), recurrent neural networks with long short-term memory (RNN-LSTM) or the combination of them. In this paper, we investigate the intrinsic characteristics of text recognition, and inspired by human cognition mechanisms in reading texts, we propose a scene text recognition method with character models on convolutional feature map. The method simultaneously detects and recognizes characters by sliding the text line image with character models, which are learned end-to-end on text line images labeled with text transcripts. The character classifier outputs on the sliding windows are normalized and decoded with Connectionist Temporal Classification (CTC) based algorithm. Compared to previous methods, our method has a number of appealing properties: (1) It avoids the difficulty of character segmentation which hinders the performance of segmentation-based recognition methods; (2) The model can be trained simply and efficiently because it avoids gradient vanishing/exploding in training RNN-LSTM based models; (3) It bases on character models trained free of lexicon, and can recognize unknown words. (4) The recognition process is highly parallel and enables fast recognition. Our experiments on several challenging English and Chinese benchmarks, including the IIIT-5K, SVT, ICDAR03/13 and TRW15 datasets, demonstrate that the proposed method yields superior or comparable performance to state-of-the-art methods while the model size is relatively small.

Motivation & Objective

• Motivate robust scene text recognition by mimicking human reading with sliding windows and fixations.
• Develop a convolutional character model that operates on sliding windows to detect and recognize characters.
• Train end-to-end on weakly labeled word images without character-level annotations.
• Utilize CTC transcription to convert per-window predictions into final text sequences.
• Demonstrate competitive performance on English and Chinese scene text benchmarks with a compact model.

Proposed method

• Use a three-part framework: sliding window feature extraction, a convolutional character classifier, and a transcription layer.
• Employ a 15-layer CNN as the character model that takes 32x32 grayscale windows and outputs a multi-class character distribution.
• Train the network end-to-end with negative log-likelihood of the ground-truth sequence, avoiding explicit character-level labeling.
• Apply Connectionist Temporal Classification (CTC) to map per-window outputs to a transcription without alignments.
• Decoding can be naive (best path), lexicon-based, or language-model-based, with a refined CTC beam search that incorporates an LM and a pruning strategy.
• Support end-to-end training and parallelizable per-window classification for efficiency.

Experimental results

Research questions

• RQ1 Can a sliding-window character model coupled with CTC effectively recognize scene text without character-level segmentation?
• RQ2 How does the approach perform on English and Chinese scene text benchmarks compared to state-of-the-art methods?
• RQ3 What is the impact of multi-scale sliding windows and language-model integration on recognition accuracy and speed?
• RQ4 Can the model operate lexicon-free and still maintain competitive accuracy across datasets?
• RQ5 How small can the model be while preserving performance, and is it suitable for mobile deployment?

Key findings

• The method yields competitive or superior performance on English benchmarks (IC03, IC13, IIIT5k, SVT) while using a relatively small model (around 8.1M parameters in the base configuration, with reductions possible via residual nets).
• A multi-scale (n=3) sliding-window version outperforms single-scale, improving recognition by capturing more context.
• With naive decoding, the approach achieves accuracy comparable to state-of-the-art methods, and with lexicon-based or LM-based decoding it can surpass them on certain datasets (e.g., IIIT5k).
• The Chinese TRW15 results show notable improvement over prior methods, e.g., TRW15-T accuracy of 81.2% (LM) and TRW15-A of 81.7% (LM) for the base model, indicating strong cross-script generalization.
• The per-sample inference time is highly efficient (about 0.015s per sample with naive decoding on the tested hardware).
• Training on synthetic data (Synth, Synth-Ch) enables strong zero-shot performance on real-world datasets, and the model remains lexicon-free and language-model-free in the basic setup.