[Paper Review] Naive-Deep Face Recognition: Touching the Limit of LFW Benchmark or Not?

Erjin Zhou, Zhimin Cao+1 more|arXiv (Cornell University)|Jan 20, 2015

Face recognition and analysis13 references194 citations

TL;DR

This paper presents a simple deep convolutional network trained on a large web-collected face dataset (5M images, 20K individuals) that achieves 99.50% accuracy on the LFW benchmark—surpassing human-level performance. Despite this high accuracy, the system fails in real-world applications like the CHID security benchmark, where it achieves only 66% true positive rate at a 10⁻⁵ false positive rate, revealing critical gaps in data bias, low false positive requirements, and cross-factors like age and pose variation.

ABSTRACT

Face recognition performance improves rapidly with the recent deep learning technique developing and underlying large training dataset accumulating. In this paper, we report our observations on how big data impacts the recognition performance. According to these observations, we build our Megvii Face Recognition System, which achieves 99.50% accuracy on the LFW benchmark, outperforming the previous state-of-the-art. Furthermore, we report the performance in a real-world security certification scenario. There still exists a clear gap between machine recognition and human performance. We summarize our experiments and present three challenges lying ahead in recent face recognition. And we indicate several possible solutions towards these challenges. We hope our work will stimulate the community's discussion of the difference between research benchmark and real-world applications.

Motivation & Objective

• To investigate how large-scale web-collected data impacts face recognition performance.
• To evaluate whether state-of-the-art LFW performance translates to real-world applications.
• To identify key challenges in deploying face recognition systems beyond standard benchmarks.
• To propose data-centric solutions for improving robustness in real-world scenarios like security certification.

Proposed method

• Built a simple ten-layer deep convolutional neural network with a softmax output layer for multi-class classification on the Megvii Face Classification (MFC) database.
• Used the final hidden layer before softmax as the face embedding, followed by PCA for dimensionality reduction.
• Measured face similarity using L2 distance between embeddings.
• Collected and curated a large-scale web-based dataset of 5 million labeled celebrity faces with significant long-tail distribution.
• Introduced the Chinese ID (CHID) benchmark to evaluate performance under real-world security constraints, particularly low false positive rates.
• Conducted human evaluation on failed cases to compare machine and human recognition performance under strict FPR conditions.

Experimental results

Research questions

• RQ1 How does the scale and distribution of web-collected data affect face recognition performance?
• RQ2 To what extent does high LFW accuracy correlate with real-world application performance?
• RQ3 What are the key challenges that prevent state-of-the-art face recognition systems from succeeding in real-world deployment?
• RQ4 How do cross-factors such as age variation, pose, and occlusion impact system performance in domain-specific settings?

Key findings

• A simple deep learning model trained on 5 million web-collected faces achieved 99.50% accuracy on the LFW benchmark, exceeding human-level performance.
• Despite high LFW accuracy, the system achieved only 66% true positive rate at a 10⁻⁵ false positive rate on the CHID benchmark, indicating a significant performance gap in real-world security applications.
• Human evaluation on failed cases showed 90% of failures could be resolved by humans, proving machine recognition is still far from human-level robustness in real-world scenarios.
• The long-tail distribution of web-collected data—where most individuals have few training samples—significantly limits recognition performance in standard multi-class classification frameworks.
• Age variation, including both intra-person and inter-person changes, was a major failure factor in the CHID benchmark, as such data was absent in the training set.
• Existing sophisticated techniques (e.g., model ensembling, joint learning) showed diminishing returns as training data size increased, suggesting data scale is more impactful than architectural complexity.