[Paper Review] The iMet Collection 2019 Challenge Dataset

Chenyang Zhang, Christine Kaeser‐Chen+4 more|arXiv (Cornell University)|Jun 3, 2019

Advanced Image and Video Retrieval Techniques8 references12 citations

TL;DR

This paper introduces the iMet Collection 2019 Challenge Dataset, a large-scale, research-grade dataset of 155,531 professional photographs of artworks from The Metropolitan Museum of Art, annotated with 1,103 fine-grained attributes by museum experts and professional vendors. The dataset supports multi-label fine-grained visual recognition, with a long-tailed distribution and variable attribute counts per image, and was used in a Kaggle-based FGVC6 competition to advance automated artwork annotation and digital curation.

ABSTRACT

Existing computer vision technologies in artwork recognition focus mainly on instance retrieval or coarse-grained attribute classification. In this work, we present a novel dataset for fine-grained artwork attribute recognition. The images in the dataset are professional photographs of classic artworks from the Metropolitan Museum of Art, and annotations are curated and verified by world-class museum experts. In addition, we also present the iMet Collection 2019 Challenge as part of the FGVC6 workshop. Through the competition, we aim to spur the enthusiasm of the fine-grained visual recognition research community and advance the state-of-the-art in digital curation of museum collections.

Motivation & Objective

• To address the lack of high-quality, fine-grained attribute annotations in artwork recognition datasets.
• To support research in fine-grained visual categorization (FGVC) using professional museum artwork images.
• To enable automated annotation of cultural heritage collections through a large-scale, expert-verified dataset.
• To advance digital curation of museum collections using computer vision techniques.
• To establish a benchmark for multi-label artwork attribute recognition via a Kaggle competition.

Proposed method

• The dataset comprises 155,531 images from The Met’s Open Access collection, all resized to 300px shortest side and stored in PNG format.
• Annotations were sourced from two verified sources: subject matter experts (SMEs) and professional vendor annotators using a predefined taxonomy.
• A three-way split (training: 109,274; validation: 7,443; test: 38,814) was created using pivot labels to ensure label consistency across splits.
• Samples with fewer than three instances for a pivot label were discarded; labels not present in all three splits were removed.
• The evaluation metric used was the Fβ score with β=2 to prioritize recall over precision in multi-label classification.
• The challenge was hosted on Kaggle as a kernel-only competition, requiring open-sourcing of models and limiting computational resources to promote fairness and transparency.

Experimental results

Research questions

• RQ1 How can fine-grained attribute recognition be advanced in the context of cultural heritage collections?
• RQ2 What is the impact of expert-curated, research-grade annotations on multi-label artwork classification performance?
• RQ3 How do long-tailed distributions and variable attribute counts per image affect model generalization in artwork recognition?
• RQ4 Can a Kaggle-based, kernel-only competition model effectively drive innovation in automated artwork annotation with fair resource access?
• RQ5 What role can high-quality, multi-attribute datasets play in advancing digital curation of museum collections?

Key findings

• The iMet Collection 2019 dataset contains 155,531 images annotated with 1,103 distinct attributes, with the most frequent labels being 'Men' (19,974 training samples), 'Women' (14,283), and 'French' (13,549).
• The dataset exhibits a long-tailed distribution, with the least frequent attribute 'Afgan' having only one training sample.
• The number of attributes per image ranges from 1 to 11, with most samples having 2 to 4 attributes, and one sample having 11 attributes.
• The competition attracted over 500 competitors from 435 teams, with results tracked on a public leaderboard and models required to be open-sourced.
• The dataset was used in the first kernel-only computer vision competition on Kaggle, promoting algorithm transparency and resource fairness.
• The authors plan to extend the dataset with additional semantic groups such as medium and size, and support future tasks like instance detection, segmentation, and caption generation.