[Paper Review] Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks

Ian Goodfellow, Yaroslav Bulatov+3 more|arXiv (Cornell University)|Dec 20, 2013

Handwritten Text Recognition TechniquesComputer Science435 citations

TL;DR

This paper proposes a deep convolutional neural network that end-to-end localizes, segments, and recognizes multi-digit numbers directly from Street View images, achieving 97.84% accuracy on per-digit recognition and over 96% on full street number recognition, with 99.8% accuracy on the hardest reCAPTCHA puzzles, demonstrating human-level performance on key tasks.

ABSTRACT

Recognizing arbitrary multi-character text in unconstrained natural photographs is a hard problem. In this paper, we address an equally hard sub-problem in this domain viz. recognizing arbitrary multi-digit numbers from Street View imagery. Traditional approaches to solve this problem typically separate out the localization, segmentation, and recognition steps. In this paper we propose a unified approach that integrates these three steps via the use of a deep convolutional neural network that operates directly on the image pixels. We employ the DistBelief implementation of deep neural networks in order to train large, distributed neural networks on high quality images. We find that the performance of this approach increases with the depth of the convolutional network, with the best performance occurring in the deepest architecture we trained, with eleven hidden layers. We evaluate this approach on the publicly available SVHN dataset and achieve over $96\%$ accuracy in recognizing complete street numbers. We show that on a per-digit recognition task, we improve upon the state-of-the-art, achieving $97.84\%$ accuracy. We also evaluate this approach on an even more challenging dataset generated from Street View imagery containing several tens of millions of street number annotations and achieve over $90\%$ accuracy. To further explore the applicability of the proposed system to broader text recognition tasks, we apply it to synthetic distorted text from reCAPTCHA. reCAPTCHA is one of the most secure reverse turing tests that uses distorted text to distinguish humans from bots. We report a $99.8\%$ accuracy on the hardest category of reCAPTCHA. Our evaluations on both tasks indicate that at specific operating thresholds, the performance of the proposed system is comparable to, and in some cases exceeds, that of human operators.

Motivation & Objective

• To develop an end-to-end system that unifies localization, segmentation, and recognition of multi-digit numbers in unconstrained street-level images.
• To improve upon traditional pipeline-based approaches by eliminating separate stages for localization and segmentation.
• To evaluate the model’s performance on real-world datasets, including the SVHN dataset and a large-scale Street View dataset with tens of millions of annotations.
• To assess the model’s generalization to synthetic, distorted text by applying it to reCAPTCHA puzzles.
• To determine whether deep architectures can achieve human-level performance on complex, real-world OCR tasks.

Proposed method

• A deep convolutional neural network with eleven hidden layers is trained end-to-end on raw pixel data, directly mapping images to digit sequences.
• The network uses a novel output layer that models sequences as conditionally independent digits with a probabilistic framework for sequence prediction.
• Training is performed using the DistBelief framework to scale large, distributed neural networks across multiple machines.
• The model leverages hierarchical feature learning, where early layers perform localization and segmentation, and deeper layers focus on recognition.
• The architecture is designed to handle variable-length sequences up to a bounded maximum length N, with each digit classified using a separate weight matrix.
• A sliding window decoding strategy is explored as a potential solution for longer sequences to improve statistical efficiency.

Experimental results

Research questions

• RQ1 Can a deep convolutional neural network effectively perform joint localization, segmentation, and recognition of multi-digit numbers in unconstrained street-level imagery?
• RQ2 Does increasing network depth significantly improve performance on multi-digit number recognition compared to shallow architectures?
• RQ3 Can a unified deep learning model achieve human-level performance on challenging OCR tasks such as distorted reCAPTCHA puzzles?
• RQ4 To what extent does model performance depend on the depth and representational capacity of the network, rather than just parameter count?
• RQ5 How does the model scale to large-scale, real-world datasets with tens of millions of annotated street numbers?

Key findings

• The model achieves 97.84% accuracy on the per-digit recognition task, surpassing the state-of-the-art at the time.
• On the full street number recognition task using the SVHN dataset, the model attains over 96% accuracy.
• On a large-scale dataset derived from Street View imagery with tens of millions of annotations, the model achieves over 90% accuracy.
• On the hardest category of reCAPTCHA puzzles, the model achieves 99.8% transcription accuracy, exceeding human-level performance at specific operating thresholds.
• Performance increases with network depth, and deeper architectures significantly outperform wider shallow models, which suffer from overfitting.
• The model successfully transcribed nearly 100 million street numbers from Street View imagery at operator-level accuracy, significantly improving geocoding quality in multiple countries.