[Paper Review] Optimal Boxes: Boosting End-to-End Scene Text Recognition by Adjusting Annotated Bounding Boxes via Reinforcement Learning

Jingqun Tang, Wenming Qian+4 more|arXiv (Cornell University)|Jul 25, 2022

Handwritten Text Recognition Techniques4 citations

TL;DR

This paper proposes Box Adjuster, a reinforcement learning method that optimizes annotated text bounding boxes for improved end-to-end scene text recognition. By training a BoxDQN agent with a recognition-based reward to adjust box shapes using foreground, background, and coordinate features, the method boosts F-Score by 2.0% on average across benchmarks and 4.6% on domain adaptation tasks, with no inference cost.

ABSTRACT

Text detection and recognition are essential components of a modern OCR system. Most OCR approaches attempt to obtain accurate bounding boxes of text at the detection stage, which is used as the input of the text recognition stage. We observe that when using tight text bounding boxes as input, a text recognizer frequently fails to achieve optimal performance due to the inconsistency between bounding boxes and deep representations of text recognition. In this paper, we propose Box Adjuster, a reinforcement learning-based method for adjusting the shape of each text bounding box to make it more compatible with text recognition models. Additionally, when dealing with cross-domain problems such as synthetic-to-real, the proposed method significantly reduces mismatches in domain distribution between the source and target domains. Experiments demonstrate that the performance of end-to-end text recognition systems can be improved when using the adjusted bounding boxes as the ground truths for training. Specifically, on several benchmark datasets for scene text understanding, the proposed method outperforms state-of-the-art text spotters by an average of 2.0% F-Score on end-to-end text recognition tasks and 4.6% F-Score on domain adaptation tasks.

Motivation & Objective

• To address the inconsistency between tight, annotated text bounding boxes and deep representations used in text recognition models.
• To improve end-to-end scene text recognition performance by learning optimal bounding box shapes that maximize recognition confidence.
• To reduce domain shift in cross-domain settings such as synthetic-to-real by refining ground-truth annotations.
• To develop a general-purpose, training-phase preprocessing method that requires no additional inference cost.
• To extend the approach to arbitrarily-shaped text using Bezier curve representations.

Proposed method

• A reinforcement learning framework, BoxDQN, is trained to adjust bounding box coordinates (x, y, width, height, rotation) to maximize recognition confidence.
• The BoxDQN agent uses a Feature Fusion Module (FFM) that integrates foreground image patches, background context, and spatial coordinates as input.
• A recognition-based reward signal is computed using a pre-trained text recognizer to guide policy learning.
• The method is applied only during dataset preparation, not at inference time, ensuring zero runtime overhead.
• For arbitrary-shaped text, Bezier curves with fixed control points (8) are used to represent variable-length polygonal boxes, enabling optimization.
• The BoxDQN is trained with a DQN algorithm using experience replay and target networks, with hyperparameters tuned via ablation.

Experimental results

Research questions

• RQ1 Can adjusting annotated bounding boxes improve end-to-end scene text recognition performance beyond using tight ground-truth boxes?
• RQ2 How does the proposed reinforcement learning-based box adjustment method reduce domain shift in synthetic-to-real transfer learning?
• RQ3 What is the optimal input representation for the box adjustment agent—foreground only, or including background and coordinates?
• RQ4 How many training iterations are needed for the BoxDQN to converge to optimal box adjustments?
• RQ5 Can the method be extended to arbitrarily-shaped text using a fixed-representation of Bezier curves?

Key findings

• The proposed Box Adjuster improves average F-Score by 2.0% on standard end-to-end text recognition benchmarks compared to state-of-the-art methods.
• On domain adaptation tasks (e.g., synthetic-to-real), the method achieves a 4.6% relative improvement in F-Score over existing approaches.
• The Feature Fusion Module (FFM) that includes background and coordinate features leads to a 0.6% F-Score gain over DQN using only foreground images.
• The optimal number of training iterations for BoxDQN is 20, achieving the best performance with minimal computational cost.
• On the TotalText dataset with arbitrary-shaped text, BoxDQN improves F-Score from 61.5 to 63.8 when using Bezier curve representations.
• Ablation studies confirm that the method generalizes across different recognition and detection backbones, showing consistent gains.