[Paper Review] Multi-Granularity Prediction with Learnable Fusion for Scene Text Recognition

Cheng Da, Peng Wang+1 more|arXiv (Cornell University)|Jul 25, 2023

Handwritten Text Recognition Techniques4 citations

TL;DR

This paper proposes MGP-STR, a vision transformer-based scene text recognition model that enhances performance through multi-granularity prediction—integrating character, BPE, and WordPiece subword representations—without an explicit language model. It introduces a learnable fusion strategy (LFS) that measures cross-modal similarity between predicted text and image features, achieving state-of-the-art accuracy of 94% on standard benchmarks like IIIT-5K, ICDAR 2015, SVT, and CUTE.

ABSTRACT

Due to the enormous technical challenges and wide range of applications, scene text recognition (STR) has been an active research topic in computer vision for years. To tackle this tough problem, numerous innovative methods have been successively proposed, and incorporating linguistic knowledge into STR models has recently become a prominent trend. In this work, we first draw inspiration from the recent progress in Vision Transformer (ViT) to construct a conceptually simple yet functionally powerful vision STR model, which is built upon ViT and a tailored Adaptive Addressing and Aggregation (A$^3$) module. It already outperforms most previous state-of-the-art models for scene text recognition, including both pure vision models and language-augmented methods. To integrate linguistic knowledge, we further propose a Multi-Granularity Prediction strategy to inject information from the language modality into the model in an implicit way, \ie, subword representations (BPE and WordPiece) widely used in NLP are introduced into the output space, in addition to the conventional character level representation, while no independent language model (LM) is adopted. To produce the final recognition results, two strategies for effectively fusing the multi-granularity predictions are devised. The resultant algorithm (termed MGP-STR) is able to push the performance envelope of STR to an even higher level. Specifically, MGP-STR achieves an average recognition accuracy of $94\%$ on standard benchmarks for scene text recognition. Moreover, it also achieves state-of-the-art results on widely-used handwritten benchmarks as well as more challenging scene text datasets, demonstrating the generality of the proposed MGP-STR algorithm. The source code and models will be available at: \url{https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/OCR/MGP-STR}.

Motivation & Objective

• To improve scene text recognition (STR) performance by implicitly integrating linguistic knowledge without relying on separate language models.
• To address the limitations of character-level-only prediction by introducing subword-level representations (BPE and WordPiece) for enhanced robustness to noise and deformation.
• To design a learnable fusion mechanism that dynamically combines multi-granularity predictions based on image-text similarity, improving final recognition accuracy.
• To develop a pure-vision STR model based on Vision Transformer and an adaptive attention module (A3) that outperforms existing vision-only and language-augmented methods.
• To achieve state-of-the-art performance across diverse benchmarks, including challenging real-world and handwritten text datasets, while avoiding iterative refinement for faster inference.

Proposed method

• Constructs a vision-only STR model based on Vision Transformer (ViT) enhanced with a tailored Adaptive Addressing and Aggregation (A3) module for improved feature representation.
• Introduces Multi-Granularity Prediction (MGP) by expanding the output space to include character-level, BPE-level, and WordPiece-level predictions, enabling multi-task learning without an external language model.
• Employs a Confidence-based Fusion Strategy (CFS) that fuses predictions using confidence scores from each branch, providing a simple yet effective fusion mechanism.
• Proposes a Learnable Fusion Strategy (LFS) that uses a trainable module to compute cross-modal similarity between predicted text and image features, inspired by CLIP, enabling dynamic, context-aware fusion.
• Trains the model end-to-end with three separate loss functions for character, BPE, and WordPiece predictions, allowing linguistic priors to be implicitly learned through multi-task optimization.
• Utilizes the Image A3 module to generate sparse, attention-based feature maps that highlight salient text regions, functioning as a global pooling mechanism for image-text matching in LFS.

Experimental results

Research questions

• RQ1 Can multi-granularity prediction—incorporating subword representations (BPE and WordPiece)—improve scene text recognition performance without using an explicit language model?
• RQ2 How does learnable fusion based on image-text similarity compare to confidence-based fusion in terms of recognition accuracy and robustness?
• RQ3 To what extent does the proposed MGP-STR model generalize across diverse benchmarks, including standard, handwritten, and challenging real-world scene text datasets?
• RQ4 Does the absence of iterative refinement in MGP-STR lead to faster inference while maintaining state-of-the-art accuracy compared to prior methods?
• RQ5 Can the model’s performance be further enhanced by leveraging the compositional nature of language through multi-granularity prediction?

Key findings

• MGP-STR with the Learnable Fusion Strategy (LFS) achieves a state-of-the-art average recognition accuracy of 94% on standard scene text benchmarks, including IIIT-5K, ICDAR 2015, SVT, and CUTE.
• The model outperforms both pure-vision and language-augmented STR methods, even without an explicit language model, demonstrating the effectiveness of implicit linguistic knowledge integration.
• The Learnable Fusion Strategy (LFS) significantly improves performance over Confidence-based Fusion (CFS), indicating that similarity-based fusion better aligns predicted text with visual features.
• The MGP-STR model achieves state-of-the-art results on handwritten text benchmarks (IAM, CVL, RIMES), showing strong generalization to cursive and low-quality text.
• The model also achieves SOTA performance on challenging datasets like ArT, COCO-Text, and Uber-Text, confirming its robustness to complex real-world scenarios.
• Attention maps from the Image A3 module are sparse but focused on relevant text regions, indicating that the module acts as an effective global pooling mechanism for image-text matching.