[Paper Review] L2-constrained Softmax Loss for Discriminative Face Verification

Rajeev Ranjan, Carlos D. Castillo+1 more|arXiv (Cornell University)|Mar 28, 2017

Face recognition and analysis279 citations

TL;DR

The paper adds an L2-norm constraint to the softmax loss so that features lie on a hypersphere, significantly boosting face verification performance on IJB-A, LFW, and YouTube Faces datasets.

ABSTRACT

In recent years, the performance of face verification systems has significantly improved using deep convolutional neural networks (DCNNs). A typical pipeline for face verification includes training a deep network for subject classification with softmax loss, using the penultimate layer output as the feature descriptor, and generating a cosine similarity score given a pair of face images. The softmax loss function does not optimize the features to have higher similarity score for positive pairs and lower similarity score for negative pairs, which leads to a performance gap. In this paper, we add an L2-constraint to the feature descriptors which restricts them to lie on a hypersphere of a fixed radius. This module can be easily implemented using existing deep learning frameworks. We show that integrating this simple step in the training pipeline significantly boosts the performance of face verification. Specifically, we achieve state-of-the-art results on the challenging IJB-A dataset, achieving True Accept Rate of 0.909 at False Accept Rate 0.0001 on the face verification protocol. Additionally, we achieve state-of-the-art performance on LFW dataset with an accuracy of 99.78%, and competing performance on YTF dataset with accuracy of 96.08%.

Motivation & Objective

• Motivate training softmax-based face verification to directly optimize verification-level separation between same- and different-identity pairs.
• Introduce a simple L2-norm constraint to enforce constant feature magnitude across samples.
• Demonstrate that the constrained loss yields better verification margins in normalized space without adding complex supervision.
• Provide practical implementation details and analysis of the scaling parameter alpha and its bounds.

Proposed method

• Introduce an L2-normalize and scale layer after the penultimate network layer to enforce a fixed feature radius alpha.
• Constrain the feature descriptor f(x) to have L2 norm equal to alpha: ||f(x_i)||_2 = alpha for all i in a mini-batch.
• Optimize the standard softmax loss with this constraint (L2-softmax) to maximize cosine similarity for positives and minimize it for negatives in the normalized space.
• Derive and discuss gradient propagation through the L2-normalize and scale layers (including alpha) during training.
• Provide theoretical guidance on choosing alpha, including a lower bound a_low derived from class count C and feature dimension D.

Experimental results

Research questions

• RQ1 Does constraining feature norms via L2-softmax improve discrimination for face verification compared to regular softmax?
• RQ2 How does the scaling parameter alpha affect performance across datasets and network architectures?
• RQ3 Can the L2-softmax approach be effectively combined with existing auxiliary losses (e.g., center loss) or metric-learning methods?
• RQ4 Is the improvement consistent across varying training set sizes and network backbones?

Key findings

• LFW accuracy improves to 99.78% using RX101 with L2-softmax, exceeding many baselines.
• On YouTube Faces, L2-softmax with RX101 achieves 96.08% accuracy, competitive with state-of-the-art.
• On the challenging IJB-A 1:1 verification protocol, L2-softmax with alpha=12–32 yields TAR at FAR=0.0001 up to 0.734–0.744 range depending on alpha, surpassing softmax baseline.
• Combining L2-softmax with auxiliary losses (e.g., center loss) provides further gains, e.g., center loss + L2-softmax achieving 99.33% on LFW.
• Using TPE with L2-softmax on IJB-A achieves a TAR of 0.909 at FAR=0.0001, surpassing prior records.
• The method demonstrates consistent improvements with multiple DCNN backbones (Face-Resnet, All-In-One Face, ResNet-101, ResNeXt-101).