[Paper Review] Is Medieval Distant Viewing Possible? : Extending and Enriching Annotation of Legacy Image Collections using Visual Analytics

Christofer Meinecke, Estelle Guéville+2 more|arXiv (Cornell University)|Aug 20, 2022

Image Retrieval and Classification Techniques4 citations

TL;DR

This paper presents a visual analytics framework that enables medievalists to unify, enrich, and hierarchically organize fragmented annotations from two legacy medieval manuscript collections (Mandragore and Initiale). By integrating word embeddings, image embeddings, co-occurrence analysis, and interactive visualizations, the system supports collaborative, semi-supervised labeling and the creation of a high-quality, domain-specific label hierarchy suitable for supervised machine learning in cultural heritage applications.

ABSTRACT

Distant viewing approaches have typically used image datasets close to the contemporary image data used to train machine learning models. To work with images from other historical periods requires expert annotated data, and the quality of labels is crucial for the quality of results. Especially when working with cultural heritage collections that contain myriad uncertainties, annotating data, or re-annotating, legacy data is an arduous task. In this paper, we describe working with two pre-annotated sets of medieval manuscript images that exhibit conflicting and overlapping metadata. Since a manual reconciliation of the two legacy ontologies would be very expensive, we aim (1) to create a more uniform set of descriptive labels to serve as a "bridge" in the combined dataset, and (2) to establish a high quality hierarchical classification that can be used as a valuable input for subsequent supervised machine learning. To achieve these goals, we developed visualization and interaction mechanisms, enabling medievalists to combine, regularize and extend the vocabulary used to describe these, and other cognate, image datasets. The visual interfaces provide experts an overview of relationships in the data going beyond the sum total of the metadata. Word and image embeddings as well as co-occurrences of labels across the datasets, enable batch re-annotation of images, recommendation of label candidates and support composing a hierarchical classification of labels.

Motivation & Objective

• Address the challenge of inconsistent, incomplete, and conflicting metadata across legacy cultural heritage image collections, particularly in medieval manuscript databases.
• Overcome the limitations of existing image classification hierarchies like ImageNet, which are ill-suited for religious and historical imagery with domain-specific semantics.
• Enable domain experts (medievalists) to collaboratively construct a unified, high-quality, and semantically meaningful label hierarchy for medieval illuminations.
• Support scalable and interactive annotation of large image collections by combining machine learning recommendations with expert feedback in a visual analytics interface.
• Facilitate cross-collection discoverability and interoperability by aligning divergent vocabularies and metadata schemas across institutions.

Proposed method

• Developed a visual analytics system that integrates word embeddings (e.g., from NLP models) to suggest and align semantic label candidates across collections.
• Utilized image embeddings (e.g., from convolutional neural networks) to enable visual similarity-based recommendations during annotation.
• Incorporated co-occurrence analysis of labels across images to suggest semantically related annotations and support hierarchical organization.
• Implemented a semi-supervised learning loop where user feedback dynamically updates visual representations (e.g., UMAP projections) and label embeddings.
• Provided interactive visualizations of label hierarchies, embedding spaces, and annotation confidence to support expert decision-making.
• Enabled batch annotation of multiple images using shared label suggestions derived from embeddings and co-occurrence patterns.

Experimental results

Research questions

• RQ1 How can fragmented and inconsistent annotations from multiple medieval manuscript databases be unified into a coherent, shared vocabulary?
• RQ2 To what extent can visual analytics and embedding-based recommendations improve the efficiency and consistency of expert annotation in cultural heritage collections?
• RQ3 Can a hierarchical label structure be co-constructed by domain experts and machine learning tools in a way that reflects both scholarly conventions and visual semantics?
• RQ4 How can visual analytics support the detection and resolution of annotation conflicts and semantic drift across legacy metadata?
• RQ5 What role can interactive feedback loops between users and embedding spaces play in refining label hierarchies and improving downstream machine learning performance?

Key findings

• The visual analytics system successfully enabled medievalists to unify and regularize conflicting metadata from Mandragore and Initiale, reducing lexical and semantic inconsistencies.
• Interactive recommendations based on word and image embeddings significantly reduced the time and cognitive load required for manual annotation of large image sets.
• The co-occurrence analysis of labels revealed meaningful semantic relationships that informed the construction of a coherent, domain-specific hierarchical classification.
• User feedback dynamically updated the visual representation of the embedding space, allowing experts to explore and refine label relationships iteratively.
• The system demonstrated strong potential for scalability and generalization, with a foundation for extending the label hierarchy to other medieval art genres and collections.
• The framework supports collaborative annotation and conflict resolution, with visual cues to represent inter-annotator disagreement and collective knowledge building.