[Paper Review] Balanced Alignment for Face Recognition: A Joint Learning Approach

Huawei Wei, Peng Lu+1 more|arXiv (Cornell University)|Mar 23, 2020

Face recognition and analysis35 references4 citations

TL;DR

This paper proposes a joint learning framework for face recognition that balances alignment strength to avoid over-alignment, which harms recognition accuracy. By performing alignment on feature maps with learnable, adaptive landmarks and a differentiable alignment loss, the method achieves state-of-the-art performance on LFW, YTF, and MegaFace, especially under large pose variations.

ABSTRACT

Face alignment is crucial for face recognition and has been widely adopted. However, current practice is too simple and under-explored. There lacks an understanding of how important face alignment is and how it should be performed, for recognition. This work studies these problems and makes two contributions. First, it provides an in-depth and quantitative study of how alignment strength affects recognition accuracy. Our results show that excessive alignment is harmful and an optimal balanced point of alignment is in need. To strike the balance, our second contribution is a novel joint learning approach where alignment learning is controllable with respect to its strength and driven by recognition. Our proposed method is validated by comprehensive experiments on several benchmarks, especially the challenging ones with large pose.

Motivation & Objective

• To investigate the quantitative impact of alignment strength on face recognition accuracy.
• To address the lack of understanding and control in current face alignment methods, which are either hand-crafted or unsupervised.
• To develop a controllable, recognition-driven alignment mechanism that avoids over-alignment and preserves identity-relevant features.
• To improve performance on challenging benchmarks with large pose variations through joint optimization of recognition and alignment.

Proposed method

• Introduces a novel alignment loss that explicitly supervises landmark deformation, enabling control over alignment strength via loss ratio tuning.
• Performs spatial transformation on feature maps instead of raw input images, preserving fine-grained facial details critical for recognition.
• Learns a deformable template with adaptive landmark weights, allowing the network to discover optimal landmark positions for recognition.
• Automatically learns the template from data rather than relying on a fixed, manually designed template.
• Extends Spatial Transformer Networks (STN) with end-to-end, recognition-driven training to jointly optimize alignment and recognition.
• Uses a differentiable warping operation on feature maps to align geometric variations while maintaining identity-invariant features.

Experimental results

Research questions

• RQ1 How does the strength of face alignment affect recognition accuracy, and is there an optimal balance?
• RQ2 Can alignment be made controllable and driven by recognition performance rather than fixed templates or unsupervised objectives?
• RQ3 Does performing alignment on feature maps instead of input images preserve more identity-relevant information?
• RQ4 Can a learnable, adaptive landmark template outperform fixed templates in recognition tasks?
• RQ5 How does the proposed method perform on challenging benchmarks with large pose variations?

Key findings

• Recognition accuracy increases with alignment strength up to a point, then decreases, indicating that excessive alignment harms performance due to distortion of facial geometry.
• The proposed method achieves 99.85% rank-1 accuracy on LFW, 96.63% on YTF, and 98.62% on MegaFace, outperforming baseline and existing alignment methods.
• Using a learnable template instead of a fixed one improves recognition accuracy, with the learned template showing looser landmark distribution that may enhance feature distinguishability.
• Alignment on feature maps yields better performance than on input images, as it preserves subtle facial details critical for identity.
• The ablation study confirms that both the alignment loss and feature map alignment contribute significantly to performance gains.
• The method achieves state-of-the-art results comparable to ArcFace and CosFace, especially on challenging large-pose datasets.