[Paper Review] Revisiting Scene Text Recognition: A Data Perspective

Qing Jiang, Jiapeng Wang+3 more|arXiv (Cornell University)|Jul 17, 2023

Handwritten Text Recognition Techniques4 citations

TL;DR

This paper re-evaluates scene text recognition (STR) from a data-centric perspective, revealing that while accuracy has saturated on standard benchmarks, real-world performance remains poor due to data distribution shifts. By introducing Union14M—a large-scale real-world dataset with 4M labeled and 10M unlabeled images—it identifies seven open challenges and demonstrates that self-supervised pre-training on the unlabeled data significantly boosts STR robustness, achieving state-of-the-art performance via a Vision Transformer-based model (MAERec).

ABSTRACT

This paper aims to re-assess scene text recognition (STR) from a data-oriented perspective. We begin by revisiting the six commonly used benchmarks in STR and observe a trend of performance saturation, whereby only 2.91% of the benchmark images cannot be accurately recognized by an ensemble of 13 representative models. While these results are impressive and suggest that STR could be considered solved, however, we argue that this is primarily due to the less challenging nature of the common benchmarks, thus concealing the underlying issues that STR faces. To this end, we consolidate a large-scale real STR dataset, namely Union14M, which comprises 4 million labeled images and 10 million unlabeled images, to assess the performance of STR models in more complex real-world scenarios. Our experiments demonstrate that the 13 models can only achieve an average accuracy of 66.53% on the 4 million labeled images, indicating that STR still faces numerous challenges in the real world. By analyzing the error patterns of the 13 models, we identify seven open challenges in STR and develop a challenge-driven benchmark consisting of eight distinct subsets to facilitate further progress in the field. Our exploration demonstrates that STR is far from being solved and leveraging data may be a promising solution. In this regard, we find that utilizing the 10 million unlabeled images through self-supervised pre-training can significantly improve the robustness of STR model in real-world scenarios and leads to state-of-the-art performance.

Motivation & Objective

• To re-evaluate the current state of scene text recognition (STR) by challenging the assumption that benchmarks indicate a solved problem.
• To identify the limitations of existing STR benchmarks, which exhibit performance saturation due to their simplicity and lack of real-world complexity.
• To construct a large-scale, real-world oriented dataset (Union14M) to expose persistent challenges in real-world STR scenarios.
• To develop a challenge-driven benchmark with eight subsets to systematically evaluate STR models on seven newly identified real-world challenges.
• To explore data-centric solutions—particularly self-supervised pre-training on massive unlabeled data—for improving STR robustness in real-world settings.

Proposed method

• Consolidated Union14M, a large-scale real-world STR dataset comprising 4 million labeled images (Union14M-L) and 10 million unlabeled images (Union14M-U) from 17 public sources.
• Evaluated 13 representative STR models on Union14M-L, revealing a significant drop in accuracy (66.53%) compared to benchmarks (87.03%), indicating distribution shift issues.
• Identified seven open challenges in STR: curve text, multi-oriented text, contextless text, artistic text, multi-words text, salient text, and incomplete text.
• Constructed a challenge-driven benchmark with 400,000 generic and 9,383 challenge-specific samples from Union14M-L to enable targeted evaluation.
• Proposed MAERec, a Vision Transformer-based STR model that leverages self-supervised masked image modeling pre-training on Union14M-U to learn robust visual-textual representations.
• Fine-tuned MAERec on Union14M-L after pre-training, achieving state-of-the-art performance on both standard benchmarks and the new challenge-driven benchmark.

Figure 1: Average accuracy of STR models on six commonly used benchmarks as reported in their original papers. Models are trained with synthetic data.

Experimental results

Research questions

• RQ1 To what extent do current STR benchmarks truly reflect real-world challenges, given the observed performance saturation?
• RQ2 What are the key failure modes of state-of-the-art STR models when deployed in real-world, complex scenarios?
• RQ3 How does the performance of STR models trained on synthetic data degrade when evaluated on a large-scale real-world dataset like Union14M?
• RQ4 Which real-world text recognition challenges are under-recognized in existing research, and how can they be systematically evaluated?
• RQ5 Can self-supervised pre-training on massive unlabeled real-world images significantly improve STR generalization and robustness?

Key findings

• Performance saturation on standard benchmarks (average 87.03% accuracy) masks significant real-world performance degradation, with STR models achieving only 66.53% accuracy on Union14M-L.
• Only 2.91% of benchmark images are misrecognized by any of 13 models, but this figure drops to 1.53% when excluding human-unrecognizable samples, indicating minimal room for improvement on current benchmarks.
• The 13 models fail on seven distinct real-world challenges: curve text, multi-oriented text, contextless text, artistic text, multi-words text, salient text, and incomplete text.
• The proposed challenge-driven benchmark shows that even with real data fine-tuning, the current SOTA model achieves only 74.6% average accuracy, confirming that STR remains unsolved in real-world settings.
• Self-supervised pre-training on 10 million unlabeled images in Union14M-U improves STR model accuracy by 5.1% on the Union14M-Benchmark and achieves state-of-the-art performance with MAERec (96.2% on standard benchmarks using ViT-Base).
• MAERec outperforms other self-supervised STR methods despite using less data, highlighting the importance of data quality and distribution in self-supervised pre-training.

Figure 2: Error analysis on the six STR benchmarks.