[Paper Review] Large-Scale Historical Watermark Recognition: dataset and a new consistency-based approach

Xi Shen, Ilaria Pastrolin+5 more|arXiv (Cornell University)|Aug 27, 2019

Handwritten Text Recognition Techniques55 references4 citations

TL;DR

This paper introduces a large-scale public dataset of over 6,000 historical watermark photographs across 16,753 fine-grained classes, enabling large-scale one-shot and cross-domain recognition. It proposes a consistency-based deep learning method that improves matching accuracy by filtering local feature matches using spatial consistency, achieving 55% top-1 accuracy on a challenging 16,753-class one-shot cross-domain recognition task using drawings from the Briquet catalog.

ABSTRACT

Historical watermark recognition is a highly practical, yet unsolved challenge for archivists and historians. With a large number of well-defined classes, cluttered and noisy samples, different types of representations, both subtle differences between classes and high intra-class variation, historical watermarks are also challenging for pattern recognition. In this paper, overcoming the difficulty of data collection, we present a large public dataset with more than 6k new photographs, allowing for the first time to tackle at scale the scenarios of practical interest for scholars: one-shot instance recognition and cross-domain one-shot instance recognition amongst more than 16k fine-grained classes. We demonstrate that this new dataset is large enough to train modern deep learning approaches, and show that standard methods can be improved considerably by using mid-level deep features. More precisely, we design both a matching score and a feature fine-tuning strategy based on filtering local matches using spatial consistency. This consistency-based approach provides important performance boost compared to strong baselines. Our model achieves 55% top-1 accuracy on our very challenging 16,753-class one-shot cross-domain recognition task, each class described by a single drawing from the classic Briquet catalog. In addition to watermark classification, we show our approach provides promising results on fine-grained sketch-based image retrieval.

Motivation & Objective

• To address the lack of large-scale, curated, public datasets for fine-grained historical watermark recognition.
• To enable practical large-scale one-shot and cross-domain recognition of watermarks in historical documents.
• To overcome challenges posed by high intra-class variation, subtle inter-class differences, and diverse representations (drawings, photos, syntheses).
• To improve deep learning performance on watermark recognition by explicitly modeling spatial consistency in local feature matches.
• To demonstrate the feasibility and practical utility of automated watermark recognition for historians and archivists.

Proposed method

• The method uses mid-level deep features extracted from convolutional neural networks to enable fine-grained matching across diverse representations.
• A spatial consistency-based matching score is proposed, which filters unreliable local matches by verifying geometric consistency across feature correspondences.
• The approach includes a feature fine-tuning strategy that adapts features during training using spatially consistent matches to improve discriminative power.
• The method is applied at both inference and training time, enhancing robustness to clutter, noise, and appearance variation.
• The framework supports both one-shot instance recognition and cross-domain recognition, using a single reference drawing per class.
• The system leverages a novel dataset collected from public notarial archives and the Briquet catalog to train and evaluate the method.

Experimental results

Research questions

• RQ1 Can a large-scale, public, fine-grained dataset of historical watermarks enable effective one-shot and cross-domain recognition?
• RQ2 Does explicitly modeling spatial consistency in local feature matching improve performance over global deep learning baselines in watermark recognition?
• RQ3 How does the proposed method handle high intra-class variation and subtle inter-class differences in watermarks?
• RQ4 To what extent can mid-level deep features, when refined via spatial consistency, outperform standard CNN-based approaches on this fine-grained recognition task?
• RQ5 What are the main failure modes of the system, and how do they relate to image quality and watermark visibility?

Key findings

• The proposed consistency-based method achieves 55% top-1 accuracy on a 16,753-class one-shot cross-domain recognition task, significantly outperforming strong global deep learning baselines.
• The method demonstrates superior performance in ranking the correct watermark among highly similar classes, particularly when baseline methods assign similar scores to multiple candidates.
• Visual analysis shows that the proposed matching score assigns high contribution only to reliably matched, discriminative regions, while suppressing irrelevant or noisy regions.
• Feature fine-tuning improves the ranking of correct matches by increasing the contribution of accurate and discriminative local features.
• Failure cases are primarily due to strong lighting effects, stains, tears, or low visibility of watermarks, suggesting potential for improvement via targeted data augmentation.
• The method also achieves promising results on fine-grained sketch-based image retrieval, indicating broader applicability beyond watermark recognition.