[Paper Review] Clustering Millions of Faces by Identity

Charles Otto, D. Wang+1 more|arXiv (Cornell University)|Apr 4, 2016

Face recognition and analysis2 references5 citations

TL;DR

This paper proposes an approximate rank-order clustering algorithm using deep face embeddings to group millions of unlabeled face images into identities, achieving high scalability and F-measure of 0.87 on LFW and 0.27 on a 123M-image dataset. The method enables efficient clustering of massive face collections and introduces an internal per-cluster quality measure to prioritize high-confidence clusters for manual review.

ABSTRACT

In this work, we attempt to address the following problem: Given a large number of unlabeled face images, cluster them into the individual identities present in this data. We consider this a relevant problem in different application scenarios ranging from social media to law enforcement. In large-scale scenarios the number of faces in the collection can be of the order of hundreds of million, while the number of clusters can range from a few thousand to millions--leading to difficulties in terms of both run-time complexity and evaluating clustering and per-cluster quality. An efficient and effective Rank-Order clustering algorithm is developed to achieve the desired scalability, and better clustering accuracy than other well-known algorithms such as k-means and spectral clustering. We cluster up to 123 million face images into over 10 million clusters, and analyze the results in terms of both external cluster quality measures (known face labels) and internal cluster quality measures (unknown face labels) and run-time. Our algorithm achieves an F-measure of 0.87 on a benchmark unconstrained face dataset (LFW, consisting of 13K faces), and 0.27 on the largest dataset considered (13K images in LFW, plus 123M distractor images). Additionally, we present preliminary work on video frame clustering (achieving 0.71 F-measure when clustering all frames in the benchmark YouTube Faces dataset). A per-cluster quality measure is developed which can be used to rank individual clusters and to automatically identify a subset of good quality clusters for manual exploration.

Motivation & Objective

• Address the challenge of clustering millions of unlabeled face images into unknown identities, especially in large-scale scenarios like social media and law enforcement.
• Overcome scalability and accuracy limitations of traditional clustering methods (e.g., k-means, spectral clustering) when applied to massive, unconstrained face datasets.
• Develop a robust, scalable clustering pipeline that maintains high accuracy despite low image quality, unbalanced subject frequencies, and noisy or missing labels.
• Introduce a novel internal per-cluster quality measure to rank clusters by compactness and isolation, enabling efficient manual inspection of high-confidence results.
• Demonstrate the feasibility of clustering up to 123 million face images into over 10 million clusters while preserving identity structure.

Proposed method

• Leverage deep convolutional neural network (CNN) embeddings from a state-of-the-art face recognition model to generate high-quality, discriminative face representations.
• Adapt the rank-order clustering algorithm by Zhu et al. [7] using shared nearest neighbors (SNN) to compute similarity, improving robustness to noisy or low-quality face images.
• Introduce approximate nearest neighbor (ANN) search (e.g., using FAISS or similar) to scale the SNN computation to 100M+ images, reducing runtime complexity.
• Simplify the clustering procedure by avoiding iterative refinement steps, favoring a one-pass, scalable approach suitable for massive datasets.
• Design an internal cluster quality score based on intra-cluster compactness and inter-cluster separation, computed without external labels.
• Apply the quality measure to rank clusters, enabling prioritization of high-confidence clusters for downstream human validation or analysis.

Experimental results

Research questions

• RQ1 Can a rank-order clustering approach with deep features and approximate nearest neighbors scale to 100 million face images while maintaining high clustering accuracy?
• RQ2 How does the proposed approximate rank-order clustering method compare in accuracy and efficiency to k-means and spectral clustering on large-scale, unconstrained face datasets?
• RQ3 To what extent can an internal cluster quality measure effectively identify high-confidence clusters without relying on external labels?
• RQ4 How does clustering performance degrade with increasing dataset size and image quality variation, particularly when including distractor images from the web?
• RQ5 Can the method successfully group multiple images of the same identity in real-world, unconstrained data, even when images are near-duplicates or vary in pose and illumination?

Key findings

• The proposed approximate rank-order clustering algorithm achieved an F-measure of 0.87 on the Labeled Faces in the Wild (LFW) benchmark dataset, indicating strong performance on standard face clustering tasks.
• On the largest evaluated dataset—13,233 labeled LFW images augmented with 123 million unlabeled web images—the algorithm achieved an F-measure of 0.27, demonstrating feasibility in real-world, large-scale settings.
• The internal per-cluster quality measure successfully ranked clusters such that the top-ranked clusters were predominantly pure (i.e., containing only images of the same identity), with high compactness and isolation.
• Top clusters often contained near-duplicate images or cartoon faces, but the quality measure effectively prioritized clusters with actual face images of the same identity, even in noisy backgrounds.
• The method scaled to cluster 123 million face images into over 10 million clusters, with runtime performance significantly improved over exact methods through approximate nearest neighbor search.
• Video frame clustering on the YouTube Faces dataset achieved an F-measure of 0.71, indicating the method generalizes well to sequential visual data.