[Paper Review] Real-time End-to-End Video Text Spotter with Contrastive Representation Learning

Wejia Wu, Zhuang Li+6 more|arXiv (Cornell University)|Jul 18, 2022

Handwritten Text Recognition Techniques4 citations

TL;DR

CoText is a real-time, end-to-end video text spotting model that unifies text detection, tracking, and recognition using contrastive representation learning to capture long-range temporal dependencies. It achieves 72.0% IDF1 at 41.0 FPS on ICDAR2015video, outperforming prior methods by 10.5% in accuracy and 32.0 FPS in speed.

ABSTRACT

Video text spotting(VTS) is the task that requires simultaneously detecting, tracking and recognizing text in the video. Existing video text spotting methods typically develop sophisticated pipelines and multiple models, which is not friend for real-time applications. Here we propose a real-time end-to-end video text spotter with Contrastive Representation learning (CoText). Our contributions are three-fold: 1) CoText simultaneously address the three tasks (e.g., text detection, tracking, recognition) in a real-time end-to-end trainable framework. 2) With contrastive learning, CoText models long-range dependencies and learning temporal information across multiple frames. 3) A simple, lightweight architecture is designed for effective and accurate performance, including GPU-parallel detection post-processing, CTC-based recognition head with Masked RoI. Extensive experiments show the superiority of our method. Especially, CoText achieves an video text spotting IDF1 of 72.0% at 41.0 FPS on ICDAR2015video, with 10.5% and 32.0 FPS improvement the previous best method. The code can be found at github.com/weijiawu/CoText.

Motivation & Objective

• To develop a real-time, end-to-end video text spotting system that unifies detection, tracking, and recognition into a single framework.
• To address the limitations of existing multi-stage pipelines that are computationally heavy and unsuitable for real-time deployment.
• To improve modeling of long-range temporal dependencies in video sequences through contrastive representation learning.
• To design a lightweight, efficient architecture that maintains high accuracy while enabling high inference speed.

Proposed method

• CoText employs a contrastive learning strategy to learn robust temporal representations across video frames, enhancing modeling of long-range dependencies.
• It uses a CTC-based recognition head with Masked RoI to improve recognition accuracy by focusing on relevant text regions.
• A GPU-parallel detection post-processing module accelerates inference, enabling real-time performance.
• The model is trained end-to-end, jointly optimizing detection, tracking, and recognition in a single unified architecture.
• Temporal contrastive learning is applied to frame-level features to align positive clips (e.g., same text across frames) and separate negative ones.
• The lightweight design minimizes FLOPs and parameter count, supporting deployment on resource-constrained devices.

Experimental results

Research questions

• RQ1 Can a unified, end-to-end framework effectively replace multi-stage pipelines for video text spotting while maintaining real-time inference?
• RQ2 How does contrastive representation learning improve the modeling of temporal dependencies in video text spotting?
• RQ3 To what extent can a lightweight architecture maintain high accuracy while achieving real-time inference speeds?
• RQ4 What is the impact of Masked RoI and CTC-based recognition on recognition performance in a joint detection-tracking-recognition setup?

Key findings

• CoText achieves 72.0% IDF1 on the ICDAR2015video benchmark, representing a 10.5% improvement over the previous best method.
• The model runs at 41.0 FPS on ICDAR2015video, a 32.0 FPS improvement over the prior state-of-the-art.
• The use of contrastive learning significantly enhances performance by modeling long-range temporal coherence across frames.
• The GPU-parallel post-processing module effectively accelerates detection without compromising accuracy.
• The lightweight architecture enables real-time inference while maintaining high accuracy, making it suitable for practical deployment.
• The combination of Masked RoI and CTC-based recognition improves recognition robustness and precision.