[Paper Review] Face Transformer for Recognition

Yaoyao Zhong, Weihong Deng|arXiv (Cornell University)|Mar 27, 2021

Face recognition and analysis31 references45 citations

TL;DR

The paper explores using Transformer models for face recognition, introduces overlapping patch tokens to capture inter-patch information, and shows competitive results with CNNs when trained on MS-Celeb-1M.

ABSTRACT

Recently there has been a growing interest in Transformer not only in NLP but also in computer vision. We wonder if transformer can be used in face recognition and whether it is better than CNNs. Therefore, we investigate the performance of Transformer models in face recognition. Considering the original Transformer may neglect the inter-patch information, we modify the patch generation process and make the tokens with sliding patches which overlaps with each others. The models are trained on CASIA-WebFace and MS-Celeb-1M databases, and evaluated on several mainstream benchmarks, including LFW, SLLFW, CALFW, CPLFW, TALFW, CFP-FP, AGEDB and IJB-C databases. We demonstrate that Face Transformer models trained on a large-scale database, MS-Celeb-1M, achieve comparable performance as CNN with similar number of parameters and MACs. To facilitate further researches, Face Transformer models and codes are available at https://github.com/zhongyy/Face-Transformer.

Motivation & Objective

• Investigate feasibility of applying Transformer architectures to face recognition.
• Assess whether Transformer performance matches or surpasses CNNs with a similar number of parameters and MACs.
• Analyze how patch overlap affects inter-patch information capture.
• Evaluate Transformer models trained on large-scale face datasets across standard benchmarks.

Proposed method

• Adapt ViT-style Transformer by generating overlapping image patches as tokens.
• Use a trainable linear projection to map patches to a model dimension D.
• Concatenate a class token and apply standard Transformer encoder with LayerNorm and residual connections.
• Train with CosFace loss to enhance discriminative embedding.
• Compare with ResNet-100 and other Vision Transformers (ViT, T2T-ViT) under varying training datasets.
• Treat the output embedding with a cosine-margin softmax loss for verification.

Experimental results

Research questions

• RQ1 Can Transformer models be effective for face recognition compared to CNNs when trained on large-scale datasets?
• RQ2 Does overlapping patch token generation improve inter-patch information capture and recognition performance?
• RQ3 How does a Face Transformer trained on MS-Celeb-1M perform on mainstream benchmarks relative to CNN-based models with comparable complexity?

Key findings

• Face Transformer models trained on MS-Celeb-1M achieve competitive accuracy with CNNs of similar parameter count and MACs.
• Overlapping patch tokens (sliding patches) improve performance over non-overlapping ViT variants.
• Transformer models show strong results on TALFW and other benchmarks when trained on a large-scale dataset.
• Occlusion robustness of Face Transformer does not surpass ResNet-100 as occlusion increases.
• Attention analyses indicate the model attends to facial regions, validating the architectural design.