[Paper Review] Learning to Read by Spelling: Towards Unsupervised Text Recognition

Ankush Gupta, Andrea Vedaldi+1 more|arXiv (Cornell University)|Sep 23, 2018

Handwritten Text Recognition Techniques67 references4 citations

TL;DR

This paper proposes an unsupervised text recognition method that learns to read by aligning predicted text strings from images with statistically valid strings from unpaired corpora, using adversarial training to implicitly learn character identities without any paired image-text annotations. The method achieves 96.2% character accuracy and 84.8% word accuracy on real historical book scans using only unaligned images and text strings, demonstrating strong generalization and robustness to non-uniform spacing.

ABSTRACT

This work presents a method for visual text recognition without using any paired supervisory data. We formulate the text recognition task as one of aligning the conditional distribution of strings predicted from given text images, with lexically valid strings sampled from target corpora. This enables fully automated, and unsupervised learning from just line-level text-images, and unpaired text-string samples, obviating the need for large aligned datasets. We present detailed analysis for various aspects of the proposed method, namely - (1) impact of the length of training sequences on convergence, (2) relation between character frequencies and the order in which they are learnt, (3) generalisation ability of our recognition network to inputs of arbitrary lengths, and (4) impact of varying the text corpus on recognition accuracy. Finally, we demonstrate excellent text recognition accuracy on both synthetically generated text images, and scanned images of real printed books, using no labelled training examples.

Motivation & Objective

• To develop a text recognition system that operates without any paired image-text annotations, overcoming the high cost of supervised data collection.
• To investigate whether learning to spell correctly—by matching predicted strings to valid linguistic statistics—can implicitly enable accurate text recognition.
• To evaluate the method’s generalization to arbitrary-length inputs and varying text corpora, including unrelated domains like War and Peace.
• To demonstrate robust performance on real-world, challenging data such as historical printed books with non-uniform spacing and variable font widths.

Proposed method

• The method formulates text recognition as aligning the conditional distribution of predicted strings from images with the distribution of valid strings sampled from a target corpus, using adversarial training.
• A fully convolutional network predicts a sequence of characters from an input image, while a discriminator ensures the predicted strings match the statistical properties (e.g., n-grams, frequencies) of real language.
• The model is trained end-to-end using a GAN-like objective, where the generator (recognition network) improves by producing strings that the discriminator cannot distinguish from real, valid text.
• To handle non-uniform spacing and variable-width fonts in real scanned books, a skip-RNN is added to the final layer of the recognition network, providing residual updates to convolutional features.
• The training process uses only unaligned data: image-line pairs and separate, unpaired text strings from a corpus, eliminating the need for alignment or manual annotation.
• Character identities are learned implicitly through language constraints, with high-frequency characters learned earlier and low-frequency characters (e.g., 'z') learned last.

Experimental results

Research questions

• RQ1 Can text recognition be achieved without any paired image-text training data by relying solely on the statistical properties of valid text?
• RQ2 How does the length of training sequences affect convergence and recognition performance?
• RQ3 Does the frequency of characters in the corpus influence the order and quality of character learning?
• RQ4 Can a model trained on one text length generalize to inputs of significantly different lengths?
• RQ5 How does the choice of text corpus—especially one unrelated to the image content—affect recognition accuracy?

Key findings

• The model achieves 99% character accuracy and 95% word accuracy on synthetic text images after training on only unaligned image and text data.
• On real scanned book images with non-uniform spacing, the skip-RNN-enhanced model improves word accuracy from 45.0% to 84.8% and character accuracy from 85.6% to 96.2%.
• The order of character learning strongly correlates with character frequency: common words like 'to' and 'it' are learned early, while rare characters like 'v' and 'w' are learned later.
• The model generalizes well to inputs of arbitrary lengths, maintaining high performance on sequences from 3 to 48 characters, even when trained only on 24-character sequences.
• Using a corpus unrelated to the image content (e.g., Tolstoy’s War and Peace) reduces word accuracy to 92.53%, but using a related corpus (WMT newscrawl) maintains near-optimal performance (~95%).
• The confusion matrix shows that only low-frequency characters like 'z' are consistently misclassified, indicating that rare characters remain a challenge despite the unsupervised setup.