[Paper Review] Deep Learning Face Representation by Joint Identification-Verification

Yi Sun, Xiaogang Wang+1 more|arXiv (Cornell University)|Jun 18, 2014

Face recognition and analysis28 references1,789 citations

TL;DR

This paper proposes DeepID2, a deep convolutional neural network that jointly optimizes face identification and verification signals to learn robust face representations. By simultaneously increasing inter-personal variations through identification and reducing intra-personal variations via verification, the method achieves 99.15% face verification accuracy on the LFW benchmark—surpassing prior state-of-the-art by 67% in error rate reduction.

ABSTRACT

The key challenge of face recognition is to develop effective feature representations for reducing intra-personal variations while enlarging inter-personal differences. In this paper, we show that it can be well solved with deep learning and using both face identification and verification signals as supervision. The Deep IDentification-verification features (DeepID2) are learned with carefully designed deep convolutional networks. The face identification task increases the inter-personal variations by drawing DeepID2 extracted from different identities apart, while the face verification task reduces the intra-personal variations by pulling DeepID2 extracted from the same identity together, both of which are essential to face recognition. The learned DeepID2 features can be well generalized to new identities unseen in the training data. On the challenging LFW dataset, 99.15% face verification accuracy is achieved. Compared with the best deep learning result on LFW, the error rate has been significantly reduced by 67%.

Motivation & Objective

• To address the challenge of intra-personal variations (e.g., pose, lighting) and inter-personal differences in unconstrained face recognition.
• To improve generalization of learned features to new identities and tasks beyond training data.
• To investigate whether combining identification and verification supervisory signals yields superior deep face representations compared to using either signal alone.
• To develop a feature learning framework that enhances both discriminative power and robustness across diverse face conditions.
• To achieve state-of-the-art performance on the LFW benchmark using only facial region inputs, matching human-level accuracy.

Proposed method

• Trains a deep convolutional neural network using both face identification (multi-class classification of 8192 identities) and face verification (binary classification of image pairs) as supervisory signals.
• Leverages a joint loss function combining identification cross-entropy loss and a triplet-like verification loss (L2 norm) to optimize feature learning.
• Extracts DeepID2 features from multiple spatial patches and resolutions of each face image to capture local and global facial patterns.
• Applies PCA to reduce the dimensionality of concatenated DeepID2 features to 180 dimensions for efficient downstream processing.
• Employs a Joint Bayesian model on the PCA-reduced features for face verification, with final score fusion via SVM over seven independently selected patch sets.
• Uses a margin-based verification loss (L2+) that specifically minimizes intra-personal variations by pulling same-identity features together.

Experimental results

Research questions

• RQ1 Can joint supervision from face identification and verification tasks jointly improve deep face representation learning compared to using either signal in isolation?
• RQ2 How does the combination of identification and verification signals affect the generalization of learned features to unseen identities and verification tasks?
• RQ3 What is the relative contribution of different verification loss functions (e.g., L2, L2+, L2-, L1, cosine) in reducing intra-personal variations?
• RQ4 To what extent can complementary features from multiple face patches and resolutions improve verification accuracy when fused effectively?
• RQ5 Can a deep learning-based system achieve human-level performance on the LFW benchmark using only facial region inputs?

Key findings

• The proposed DeepID2 model achieves 99.15% face verification accuracy on the LFW dataset, setting a new state-of-the-art at the time of publication.
• The joint use of identification and verification signals reduces the error rate by 67% compared to the previous best deep learning result (97.45% accuracy).
• The L2+ verification loss, which only minimizes intra-personal distances, significantly improves performance over L2- or no verification signal, confirming its role in reducing intra-personal variation.
• The L2 norm outperforms L1, cosine, and no verification signal in verification accuracy, indicating stronger regularization for intra-personal consistency.
• Feature fusion across seven independently selected patch sets using an SVM boosts verification accuracy from 98.97% to 99.15%, demonstrating the benefit of multi-scale, multi-region feature aggregation.
• The model generalizes well to new identities and tasks, as evidenced by high performance on LFW without requiring background context or external data.