[Paper Review] MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition

Yandong Guo, Lei Zhang+3 more|arXiv (Cornell University)|Jul 27, 2016

Face recognition and analysisComputer Science17 references162 citations

TL;DR

The paper introduces a one-million-celebrities face recognition benchmark linked to a knowledge base, plus a large training set and evaluation protocol for measuring precision and coverage at scale.

ABSTRACT

In this paper, we design a benchmark task and provide the associated datasets for recognizing face images and link them to corresponding entity keys in a knowledge base. More specifically, we propose a benchmark task to recognize one million celebrities from their face images, by using all the possibly collected face images of this individual on the web as training data. The rich information provided by the knowledge base helps to conduct disambiguation and improve the recognition accuracy, and contributes to various real-world applications, such as image captioning and news video analysis. Associated with this task, we design and provide concrete measurement set, evaluation protocol, as well as training data. We also present in details our experiment setup and report promising baseline results. Our benchmark task could lead to one of the largest classification problems in computer vision. To the best of our knowledge, our training dataset, which contains 10M images in version 1, is the largest publicly available one in the world.

Motivation & Objective

• Define a large-scale face recognition benchmark that links faces to knowledge-base entity keys (one million celebrities).
• Provide datasets for training, measurement, and evaluation to enable research on disambiguation and web-scale recognition.
• Highlight the scalability challenges and baseline results to drive research in large-class classification for face recognition.
• Encourage use of outside data and future expansion of the dataset to improve real-world applicability.

Proposed method

• Construct a one-million-celebrity list from Freebase ordered by web appearance frequency and provide their Freebase MIDs.
• Assemble a large training set (~10M images for ~100K top celebrities) with cropped/aligned faces for classification-based learning.
• Design a measurement set with carefully labeled images blended with distractors to evaluate recognition performance under a fixed precision constraint.
• Define an evaluation protocol using precision and coverage with a rejection threshold to measure maximum coverage at a target precision (Pmin=0.95).
• Train a deep neural network classifier for celebrities (initially a 500-class model, then a 100K-class full model) to establish baseline performance.
• Provide baseline results showing 44.2% recognition on the measurement set at 95% precision (hard case) without excessive fine-tuning.

Experimental results

Research questions

• RQ1 Can recognition and disambiguation be scaled to one million celebrity classes while linking to knowledge-base entity keys?
• RQ2 What is the performance (precision and coverage) achievable on a large-scale celebrity recognition benchmark using web-sourced training data?
• RQ3 How does a deep neural network classifier perform as the number of classes scales to 100K and beyond in this setting?
• RQ4 What are the challenges and baseline barriers for real-world applications like image search and captioning when linking faces to rich knowledge-base properties?

Key findings

• A million-celebrity recognition task linked to knowledge-base keys is feasible and yields meaningful baselines.
• A training set of ~10 million images for ~100K top celebrities provides a strong baseline for measurement tasks.
• Baseline CNN training on 100K celebrities achieves notable precision-coverage trade-offs, illustrating the challenging scale (Hard Set: 0.052 at 99% precision, 0.442 at 95% precision; Random Set: 0.606 at 99% precision, 0.728 at 95% precision).
• The measurement set contains two images per celebrity (random and hard) blended with distractors to stress generalization and coverage.
• The authors encourage using outside data and expanding coverage to push the upper bound beyond the provided baseline.