[Paper Review] Killing Two Birds with One Stone:Efficient and Robust Training of Face Recognition CNNs by Partial FC

Xiang An, Jiankang Deng+5 more|arXiv (Cornell University)|Mar 28, 2022

Face recognition and analysis6 citations

TL;DR

This paper proposes Partial FC (PFC), a sparsely updated fully connected layer that selects only a subset of negative class centers and the positive center per training iteration to compute margin-based softmax loss. By drastically reducing computational cost, inter-class conflict, and passive updates on tail classes, PFC enables efficient, robust, and accurate training of face recognition models on large-scale datasets like WebFace42M, achieving state-of-the-art accuracy (98.00% on IJB-C) with up to 5× faster training and 50% less GPU memory usage.

ABSTRACT

Learning discriminative deep feature embeddings by using million-scale in-the-wild datasets and margin-based softmax loss is the current state-of-the-art approach for face recognition. However, the memory and computing cost of the Fully Connected (FC) layer linearly scales up to the number of identities in the training set. Besides, the large-scale training data inevitably suffers from inter-class conflict and long-tailed distribution. In this paper, we propose a sparsely updating variant of the FC layer, named Partial FC (PFC). In each iteration, positive class centers and a random subset of negative class centers are selected to compute the margin-based softmax loss. All class centers are still maintained throughout the whole training process, but only a subset is selected and updated in each iteration. Therefore, the computing requirement, the probability of inter-class conflict, and the frequency of passive update on tail class centers, are dramatically reduced. Extensive experiments across different training data and backbones (e.g. CNN and ViT) confirm the effectiveness, robustness and efficiency of the proposed PFC. The source code is available at \https://github.com/deepinsight/insightface/tree/master/recognition.

Motivation & Objective

• To address the high memory and computational cost of training deep face recognition models with millions of identities using standard fully connected (FC) layers.
• To mitigate inter-class conflict and label noise in large-scale, automatically collected datasets like WebFace42M, which degrade model performance.
• To reduce the frequency of passive updates on underrepresented tail-class centers in long-tailed distributions.
• To enable efficient training on single-GPU setups without sacrificing model accuracy.
• To achieve state-of-the-art performance on major benchmarks while significantly lowering training cost.

Proposed method

• PFC maintains a full set of class centers throughout training but updates only a sampled subset in each iteration.
• In each forward pass, the positive class center is selected based on the ground-truth label, and a random subset of negative class centers is selected for loss computation.
• The margin-based softmax loss is computed only over the selected positive and negative centers, reducing the computational complexity from O(N) to O(rN), where r is the sampling ratio.
• The full set of class centers is preserved and updated only for the sampled subset, ensuring model stability and convergence.
• A simple online abnormal inter-class filtering mechanism is applied to further improve robustness under label noise.
• The method is compatible with various backbones, including CNNs and Vision Transformers (ViT).

Experimental results

Research questions

• RQ1 Can a sparsely updated fully connected layer reduce training cost in large-scale face recognition without degrading performance?
• RQ2 How does partial sampling of negative class centers affect model robustness to label noise and inter-class conflict?
• RQ3 To what extent can PFC reduce GPU memory usage and training time on large-scale datasets like WebFace42M?
• RQ4 How does PFC perform under long-tailed data distributions compared to existing sampling or FC-free methods?
• RQ5 Can PFC achieve state-of-the-art accuracy on major benchmarks like IJB-C and CFP-FP while enabling single-node training?

Key findings

• PFC-0.1 trained on WebFace42M with ResNet100 achieves 96.19% verification accuracy on MFR-All, outperforming FC baseline and other sampling methods.
• PFC-0.008 with ResNet100 achieves 97.51% accuracy on IJB-C with near-negligible FC computation cost, demonstrating extreme efficiency.
• On a synthetic 10M identity dataset, PFC-0.1 trains five times faster than the FC baseline and uses less than half the GPU memory.
• PFC-0.2 achieves 91.96% accuracy on MFR-All under long-tailed distribution, surpassing the FC baseline by 4.52% and DCQ by 2.59%.
• PFC-0.3 achieves 98.00% accuracy on IJB-C and 99.51% on CFP-FP, setting new state-of-the-art results across multiple benchmarks.
• PFC models trained on WebFace4M with r=0.04 outperform Virtual FC by 27.47% on IJB-B and 25.33% on IJB-C, despite Virtual FC reducing parameters by over 100×.