[Paper Review] ICDAR 2019 Competition on Image Retrieval for Historical Handwritten Documents

Vincent Christlein, Anguelos Nicolaou+3 more|arXiv (Cornell University)|Dec 8, 2019

Advanced Image and Video Retrieval Techniques14 references35 citations

TL;DR

A large-scale evaluation of writer retrieval for historical documents using 20,000 images from 10,000 writers across manuscripts, letters, and charters, showing non-deep-learning methods can compete, with SIFT+Pathlet achieving top performance.

ABSTRACT

This competition investigates the performance of large-scale retrieval of historical document images based on writing style. Based on large image data sets provided by cultural heritage institutions and digital libraries, providing a total of 20 000 document images representing about 10 000 writers, divided in three types: writers of (i) manuscript books, (ii) letters, (iii) charters and legal documents. We focus on the task of automatic image retrieval to simulate common scenarios of humanities research, such as writer retrieval. The most teams submitted traditional methods not using deep learning techniques. The competition results show that a combination of methods is outperforming single methods. Furthermore, letters are much more difficult to retrieve than manuscripts.

Motivation & Objective

• Motivate writer retrieval in historical documents and enable large-scale humanities research through practical retrieval benchmarks.
• Provide a public dataset of 20,000 document images from 10,000 writers spanning manuscripts, letters, and charters.
• Assess a range of non-deep-learning and limited ML approaches for writer retrieval on diverse, noisy historical data.

Proposed method

• Use a public 20,000-image test set with 7,500 single-page writers and 12,500 pages from multi-page writers to simulate writer retrieval tasks.
• Evaluate nine submissions from three teams plus two baseline variants of a prior method, using a distance-based retrieval framework.
• Represent images with handcrafted features (LBP, SIFT-based Fisher Vectors, Pathlet) and simple classifiers or distance metrics; no full deep learning models dominate the baseline results.
• Preprocess by cropping borders and resizing images to uniform dimensions to control for trivial hashing attacks and download efficiency.
• Compute mean Average Precision (mAP) and Top-1 accuracy via leave-one-image-out cross-validation on the test set.

Experimental results

Research questions

• RQ1 What is the performance of traditional (non-deep-learning) writer retrieval methods on a large-scale historical dataset?
• RQ2 How do feature types (texture-based LBP, SIFT-based Fisher Vectors, Pathlet) and their combinations perform for writer retrieval across different document types?
• RQ3 Does the difficulty of retrieval differ between manuscripts and letters, and among dataset subsets (MSS, Letters A/B, Charters)?

Key findings

• The SCUT team achieved the highest mAP (90.6%) on the full test set using a combination of SIFT and Pathlet features.
• The best overall result was a SIFT+Pathlet combination with mAP 92.5% and Top-1 accuracy 97.4% on the full dataset.
• Baseline SRS-LBP variants performed strong results (mAP 84.0–86.8%), sometimes rivaling more complex methods.
• Traditional methods (e.g., Hinge, Co-Hinge, oBIFs) achieved competitive mAP scores (around 75–85%), indicating non-neural approaches remain effective.
• Letters are significantly more challenging to retrieve than manuscripts, with notably lower performance in the Letters A subset (mAP around 45–47%).
• Overall, no neural-network-based retrieval methods dominated in this competition; binarization used a CNN-based approach for preprocessing in one submission.