[Paper Review] Towards Accurate Scene Text Recognition with Semantic Reasoning Networks

Deli Yu, Xuan Li+4 more|arXiv (Cornell University)|Mar 27, 2020

Handwritten Text Recognition Techniques45 references56 citations

TL;DR

The paper introduces Semantic Reasoning Network (SRN) with a Global Semantic Reasoning Module (GSRM) to fuse parallel visual features and global semantic context for end-to-end scene text recognition, achieving state-of-the-art results on multiple benchmarks with faster parallel inference.

ABSTRACT

Scene text image contains two levels of contents: visual texture and semantic information. Although the previous scene text recognition methods have made great progress over the past few years, the research on mining semantic information to assist text recognition attracts less attention, only RNN-like structures are explored to implicitly model semantic information. However, we observe that RNN based methods have some obvious shortcomings, such as time-dependent decoding manner and one-way serial transmission of semantic context, which greatly limit the help of semantic information and the computation efficiency. To mitigate these limitations, we propose a novel end-to-end trainable framework named semantic reasoning network (SRN) for accurate scene text recognition, where a global semantic reasoning module (GSRM) is introduced to capture global semantic context through multi-way parallel transmission. The state-of-the-art results on 7 public benchmarks, including regular text, irregular text and non-Latin long text, verify the effectiveness and robustness of the proposed method. In addition, the speed of SRN has significant advantages over the RNN based methods, demonstrating its value in practical use.

Motivation & Objective

• Motivate the use of semantic information to aid scene text recognition beyond purely visual features.
• Develop a scalable, end-to-end trainable framework that models global semantic context in parallel.
• Propose a parallel visual attention module (PVAM) and a visual-semantic fusion decoder (VSFD) to integrate visual and semantic cues.
• Demonstrate efficiency and robustness through extensive experiments on diverse text benchmarks.

Proposed method

• Backbone with ResNet50+FPN and transformer units to capture global visual context.
• Parallel Visual Attention Module (PVAM) that produces N aligned 1-D visual features in parallel.
• Global Semantic Reasoning Module (GSRM) that uses a visual-to-semantic embedding block and a semantic reasoning block based on stacked transformer units to produce semantic features S.
• Visual-Semantic Fusion Decoder (VSFD) with a gated unit to fuse visual features G and semantic features S into final predictions.
• End-to-end trainable objective: Loss = embedding loss (L_e) + reasoning loss (L_r) + final decoding loss (L_f).

Experimental results

Research questions

• RQ1 Can global, multi-way semantic reasoning improve scene text recognition beyond one-way or sequential semantic modeling?
• RQ2 How can visual and semantic information be effectively fused in a parallel, end-to-end framework?
• RQ3 Does parallel attention for visual features plus global semantic reasoning accelerate inference while maintaining accuracy?
• RQ4 How does SRN perform on regular, irregular, and non-Latin long text benchmarks without lexicons?
• RQ5 What is the impact of GSRM configuration (number of Transformer units) and fusion strategy on performance?

Key findings

• SRN with GSRM achieves state-of-the-art performance across multiple public benchmarks including regular, irregular, and non-Latin long text datasets.
• PVAM enables parallel alignment of visual features to each target character, improving efficiency over time-dependent attention.
• GSRM provides significant gains by modeling global semantic context, with multi-way (parallel) reasoning outperforming one-way semantic reasoning variants.
• VSFD with gated fusion effectively balances visual and semantic cues, yielding robust recognition without lexicons.
• Inference speed is improved relative to RNN-based semantic models due to parallel processing, while maintaining high accuracy, especially on long text.