[Paper Review] Weakly-Supervised Arbitrary-Shaped Text Detection with Expectation-Maximization Algorithm

Mengbiao Zhao, Wei Feng+3 more|arXiv (Cornell University)|Dec 1, 2020

Handwritten Text Recognition Techniques40 references4 citations

TL;DR

This paper proposes an Expectation-Maximization (EM)-based weakly-supervised framework for arbitrary-shaped text detection, using a contour-based detector to leverage various weak supervision forms (image-level tags, coarse, loose, and tight bounding boxes). With only 10% polygon-level annotations and 90% weakly annotated data, the method achieves performance comparable to fully supervised models, while outperforming state-of-the-art methods under full supervision on CTW1500, Total-Text, and ICDAR-ArT benchmarks.

ABSTRACT

Arbitrary-shaped text detection is an important and challenging task in computer vision. Most existing methods require heavy data labeling efforts to produce polygon-level text region labels for supervised training. In order to reduce the cost in data labeling, we study weakly-supervised arbitrary-shaped text detection for combining various weak supervision forms (e.g., image-level tags, coarse, loose and tight bounding boxes), which are far easier for annotation. We propose an Expectation-Maximization (EM) based weakly-supervised learning framework to train an accurate arbitrary-shaped text detector using only a small amount of polygon-level annotated data combined with a large amount of weakly annotated data. Meanwhile, we propose a contour-based arbitrary-shaped text detector, which is suitable for incorporating weakly-supervised learning. Extensive experiments on three arbitrary-shaped text benchmarks (CTW1500, Total-Text and ICDAR-ArT) show that (1) using only 10% strongly annotated data and 90% weakly annotated data, our method yields comparable performance to state-of-the-art methods, (2) with 100% strongly annotated data, our method outperforms existing methods on all three benchmarks. We will make the weakly annotated datasets publicly available in the future.

Motivation & Objective

• To reduce the high annotation cost of polygon-level labels for arbitrary-shaped text detection by leveraging easily obtainable weak supervision forms.
• To develop a weakly-supervised learning framework that effectively combines limited strong annotations with abundant weak annotations.
• To design a contour-based detector compatible with various weak supervision formats for improved generalization and training efficiency.
• To demonstrate that weakly-supervised training can achieve performance close to fully supervised models while drastically reducing annotation effort.

Proposed method

• Proposes a contour-based arbitrary-shaped text detector that first generates text proposals and then regresses precise contours, aligning well with weak supervision forms.
• Introduces four weak supervision forms: image-level tags, coarse, loose, and tight bounding boxes, each requiring progressively less annotation time.
• Employs an EM-like optimization algorithm where the E-step estimates a probability distribution over possible latent polygon labels, and the M-step updates model weights using pseudo-labeled polygons.
• Uses a model pre-trained on a small set of strongly annotated data to initialize the EM process, improving convergence and final performance.
• Applies a confidence-weighted loss function to reduce the influence of noisy pseudo-labels during training.
• Conducts iterative EM training rounds, with performance improving up to two rounds before error accumulation halts further gains.

Experimental results

Research questions

• RQ1 Can weak supervision forms such as image-level tags and bounding boxes significantly reduce annotation cost while maintaining high detection accuracy?
• RQ2 How effective is an EM-based learning framework in leveraging both limited strong annotations and abundant weak annotations for arbitrary-shaped text detection?
• RQ3 What is the impact of different weak supervision forms (e.g., tight vs. loose bounding boxes) on model performance and annotation efficiency?
• RQ4 To what extent can a weakly-supervised model match the performance of a fully supervised model when only 10% of the data is strongly annotated?

Key findings

• Using only 10% strongly annotated data and 90% weakly annotated data, the proposed method achieves F-measure scores comparable to fully supervised state-of-the-art methods on CTW1500, Total-Text, and ICDAR-ArT.
• With 100% strongly annotated data, the method outperforms existing methods on all three benchmarks, demonstrating superior accuracy under full supervision.
• The confidence-weighted loss improves model performance by approximately 2.1–2.5% in F-measure, reducing the impact of noisy pseudo-labels.
• Performance improves over the first two EM training rounds, but plateaus or degrades in the third round due to error accumulation in pseudo-labels.
• The 'Equal Time' and 'Equal Number' weak annotation policies outperform the 'Strong' policy (full polygon labeling) by over 3.9% in F-measure, indicating better cost-effectiveness.
• Loose bounding boxes are the most cost-effective weak supervision form, achieving the best performance when 80% of the budget is spent on polygons and 20% on weak labels.