[Paper Review] Joint Visual Semantic Reasoning: Multi-Stage Decoder for Text Recognition

Ayan Kumar Bhunia, Aneeshan Sain+4 more|arXiv (Cornell University)|Jul 26, 2021

Multimodal Machine Learning Applications63 references4 citations

TL;DR

This paper proposes a multi-stage, multi-scale attentional decoder for text recognition that enhances visual-only models by integrating joint visual-semantic reasoning. By unrolling predictions stage-wise and using a straight-through Gumbel-softmax to enable end-to-end training despite discrete character predictions, the model refines initial visual estimates using contextual semantic information, achieving 84.0% WRA on IC15, a significant improvement over SOTA methods.

ABSTRACT

Although text recognition has significantly evolved over the years, state-of-the-art (SOTA) models still struggle in the wild scenarios due to complex backgrounds, varying fonts, uncontrolled illuminations, distortions and other artefacts. This is because such models solely depend on visual information for text recognition, thus lacking semantic reasoning capabilities. In this paper, we argue that semantic information offers a complementary role in addition to visual only. More specifically, we additionally utilize semantic information by proposing a multi-stage multi-scale attentional decoder that performs joint visual-semantic reasoning. Our novelty lies in the intuition that for text recognition, the prediction should be refined in a stage-wise manner. Therefore our key contribution is in designing a stage-wise unrolling attentional decoder where non-differentiability, invoked by discretely predicted character labels, needs to be bypassed for end-to-end training. While the first stage predicts using visual features, subsequent stages refine on top of it using joint visual-semantic information. Additionally, we introduce multi-scale 2D attention along with dense and residual connections between different stages to deal with varying scales of character sizes, for better performance and faster convergence during training. Experimental results show our approach to outperform existing SOTA methods by a considerable margin.

Motivation & Objective

• To address the limitations of state-of-the-art text recognition models that rely solely on visual features and fail in wild, complex scenarios such as distorted, noisy, or low-illumination images.
• To enable joint visual-semantic reasoning in text recognition by refining predictions iteratively across multiple stages, leveraging both visual and semantic context.
• To overcome the non-differentiability issue introduced by discrete character predictions in a multi-stage setting, enabling end-to-end training with gradient flow across stages.
• To improve robustness and accuracy in unconstrained text recognition by incorporating global semantic context through stage-wise refinement and multi-scale attention mechanisms.

Proposed method

• Introduces a multi-stage unrolled attentional decoder where the first stage predicts from visual features alone, and subsequent stages refine predictions using joint visual-semantic information.
• Employs a straight-through Gumbel-softmax operator to bypass the non-differentiability of argmax during training, enabling backpropagation through discrete token predictions.
• Uses multi-scale 2D convolutional attention to handle varying character sizes and improve feature alignment across scales.
• Incorporates dense and residual connections between stages to stabilize training and accelerate convergence.
• Leverages a visual-semantic reasoning module based on either LSTM or Transformer with multi-head self-attention to model long-range dependencies and enrich features with semantic context.
• Employs a pre-trained BERT-like architecture for the semantic reasoning module to improve global semantic understanding.

Figure 1 : Compared to existing attentional decoder architectures [ 53 , 33 ] , we design a novel multi-scale attention decoder for text recognition which is unpacked in a stage-wise manner . The problem of non-differentiability, due to discrete-character prediction is bypassed via straight-through

Experimental results

Research questions

• RQ1 Can joint visual-semantic reasoning significantly improve text recognition performance in wild, unconstrained scenarios where visual features alone are insufficient?
• RQ2 How can a multi-stage decoding framework effectively refine predictions using both visual and semantic context in an end-to-end differentiable manner?
• RQ3 What is the optimal number of stages for multi-stage refinement, and how does it affect model performance and convergence?
• RQ4 To what extent does incorporating multi-scale attention and residual connections improve robustness and accuracy in text recognition?
• RQ5 How does pre-training the semantic reasoning module enhance overall recognition performance?

Key findings

• The proposed multi-stage decoder achieves 84.0% word recognition accuracy (WRA) on the IC15 dataset, outperforming the previous SOTA by a significant margin.
• The model achieves a 1.7% performance gain when using separate weights for the classification and refinement heads while sharing the encoder and semantic reasoning modules, indicating optimal parameter sharing.
• The optimal number of stages is found to be two, as performance saturates at three stages, likely due to vanishing gradient effects or diminishing returns in semantic context utilization.
• The model shows superior performance even under low-data regimes, achieving 73.1% WRA with only 1M training samples, outperforming baselines and prior SOTA methods.
• Using a Transformer-based visual-semantic reasoning module with pre-training yields 84.0% WRA, a 0.9% improvement over the LSTM-based version and a 1.3% improvement over the non-pretrained Transformer baseline.
• The computational cost increases from 15.3 GFLOPs (baseline) to 22.5 GFLOPs (proposed), but this is justified by the substantial accuracy gain, especially when compared to naive stacking of deeper networks.

Figure 2 : With the extracted context-rich holistic feature ( $h_{L}$ ) and multi-scale feature maps ( $B_{L},B_{L-1},B_{L-2}$ ), a multi-stage attentional decoder predicts the character sequences in consecutive stages. Once the previous stage’s decoder completely unrolls itself across time , the cu