[Paper Review] COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Images

Andreas Veit, Tomáš Matera+3 more|arXiv (Cornell University)|Jan 26, 2016

Handwritten Text Recognition TechniquesComputer Science237 citations

TL;DR

COCO-Text introduces a large-scale, richly annotated dataset for detecting and recognizing text in natural images, expanding annotations beyond transcription to include legibility, script, and text type, and evaluates state-of-the-art photo OCR methods on this data.

ABSTRACT

This paper describes the COCO-Text dataset. In recent years large-scale datasets like SUN and Imagenet drove the advancement of scene understanding and object recognition. The goal of COCO-Text is to advance state-of-the-art in text detection and recognition in natural images. The dataset is based on the MS COCO dataset, which contains images of complex everyday scenes. The images were not collected with text in mind and thus contain a broad variety of text instances. To reflect the diversity of text in natural scenes, we annotate text with (a) location in terms of a bounding box, (b) fine-grained classification into machine printed text and handwritten text, (c) classification into legible and illegible text, (d) script of the text and (e) transcriptions of legible text. The dataset contains over 173k text annotations in over 63k images. We provide a statistical analysis of the accuracy of our annotations. In addition, we present an analysis of three leading state-of-the-art photo Optical Character Recognition (OCR) approaches on our dataset. While scene text detection and recognition enjoys strong advances in recent years, we identify significant shortcomings motivating future work.

Motivation & Objective

• Provide a large-scale, diverse dataset of text in natural scenes to advance scene text detection and recognition.
• Annotate text instances with bounding boxes and fine-grained attributes (legibility, machine printed vs handwritten, script).
• Assess current state-of-the-art photo OCR methods on the dataset and identify remaining gaps for real-world applications.

Proposed method

• Annotate MS COCO images with text regions using a multi-stage crowd-sourcing pipeline.
• Incorporate OCR outputs from multiple photo OCR systems and human annotators to detect and refine text regions.
• Classify text regions by legibility, script, and type (machine printed, handwritten, others).
• Collect transcriptions for legible text and flag illegible text during transcription iterations.
• Evaluate detection, transcription, and end-to-end performance using ICDAR-style metrics on a held-out validation set.

Experimental results

Research questions

• RQ1 How diverse is text in natural scenes when annotated on a large MS COCO-based dataset?
• RQ2 Can crowd workers together with OCR systems reliably detect and classify a wide range of text types and legibility levels in natural images?
• RQ3 What are the current limitations of state-of-the-art photo OCR methods on unconstrained scene text, particularly for illegible text and recall in detection?
• RQ4 How does context (objects in COCO) relate to the presence of text in natural images?
• RQ5 What improvements are needed to approach robust end-to-end text spotting in real-world scenarios?

Key findings

• COCO-Text contains 63,686 images with 173,589 text annotations across bounding boxes and fine-grained attributes.
• About 50% of COCO-Text images contain no text, while 2.73 text instances per image overall (5.46 per images with text).
• Text attributes include legibility (60.3% legible, 39.7% illegible), type (machine printed vs handwritten), and script (English vs not English).
• Three leading photo OCR systems achieved high precision but poor recall for text detection, especially for illegible text, highlighting substantial remaining gaps.
• Crowd annotators detected 57% of all text regions, with higher success on legible text (84%) than illegible text (39%).
• End-to-end recognition results were limited to legible machine-printed and handwritten English text, underscoring the gap between dataset richness and current OCR capabilities.