[Paper Review] Beyond the Visible: A Survey on Cross-spectral Face Recognition

David Anghelone, Cunjian Chen+2 more|arXiv (Cornell University)|Jan 12, 2022

Face recognition and analysis4 citations

TL;DR

This survey provides a comprehensive overview of cross-spectral face recognition (CFR), focusing on deep learning techniques to bridge the modality gap between visible and infrared spectra. It evaluates methods using CNNs and GANs, highlights key datasets, and identifies challenges such as spectral variation, limited data, and resolution issues, concluding with future research directions for robust, privacy-preserving biometric systems.

ABSTRACT

Cross-spectral face recognition (CFR) refers to recognizing individuals using face images stemming from different spectral bands, such as infrared versus visible. While CFR is inherently more challenging than classical face recognition due to significant variation in facial appearance caused by the modality gap, it is useful in many scenarios including night-vision biometrics and detecting presentation attacks. Recent advances in deep neural networks (DNNs) have resulted in significant improvement in the performance of CFR systems. Given these developments, the contributions of this survey are three-fold. First, we provide an overview of CFR, by formalizing the CFR problem and presenting related applications. Secondly, we discuss the appropriate spectral bands for face recognition and discuss recent CFR methods, placing emphasis on deep neural networks. In particular we describe techniques that have been proposed to extract and compare heterogeneous features emerging from different spectral bands. We also discuss the datasets that have been used for evaluating CFR methods. Finally, we discuss the challenges and future lines of research on this topic.

Motivation & Objective

• To formalize the cross-spectral face recognition (CFR) problem and clarify its distinction from traditional visible-spectrum face recognition.
• To analyze the challenges in CFR, including large intra-spectral variation, the modality gap, and limited paired training data across spectral bands.
• To review and categorize deep learning-based CFR methods, particularly those using CNNs and GANs to learn domain-invariant features and synthesize cross-spectral images.
• To evaluate existing datasets used in CFR research and identify gaps in data availability and diversity.
• To outline open research problems, including improving generalization, enhancing resolution in thermal bands, and ensuring robustness against spoofing and privacy threats.

Proposed method

• Formalize CFR as a task where face images from different spectral modalities (e.g., visible, NIR, LWIR) are matched using a deep learning model trained to minimize modality-specific appearance differences.
• Employ domain-invariant feature learning via deep neural networks, particularly using triplet loss to enforce intra-class compactness and inter-class separability across spectral domains.
• Utilize generative adversarial networks (GANs) to synthesize cross-spectral face images, with identity loss ensuring preserved identity and auxiliary losses (e.g., shape, semantic, attribution) improving realism and fidelity.
• Apply unsupervised domain adaptation techniques to align feature distributions between source (e.g., visible) and target (e.g., infrared) domains without requiring paired training samples.
• Integrate super-resolution post-processing to enhance low-resolution thermal images, especially in MWIR and LWIR bands.
• Use multi-spectral imaging, including polarimetry in LWIR, to improve recognition accuracy by capturing additional physical characteristics of facial surfaces.

Experimental results

Research questions

• RQ1 How can deep neural networks effectively learn identity-invariant features across heterogeneous spectral bands such as visible, NIR, and thermal infrared?
• RQ2 What are the most effective loss functions (e.g., triplet loss vs. contrastive loss) for training end-to-end CFR models under domain shift?
• RQ3 To what extent can GAN-based image synthesis reduce the modality gap in cross-spectral face recognition, and how do auxiliary losses improve synthesis quality?
• RQ4 How do spectral band choices (NIR, SWIR, MWIR, LWIR) affect recognition performance, and which are most suitable for real-world applications?
• RQ5 What are the key limitations of current CFR datasets, and how can future data collection improve model generalization and robustness?

Key findings

• Triplet loss outperforms contrastive loss in CFR by simultaneously minimizing intra-class variation and maximizing inter-class margin in feature space.
• GAN-based cross-spectral image synthesis shows promise in bridging the modality gap, though synthetic data has not yet surpassed state-of-the-art performance on real data.
• Near-infrared (NIR) is the preferred spectral band for reflective IR-to-Visible CFR due to better resolution and compatibility with existing systems.
• Thermal bands (MWIR/LWIR) offer advantages in low-light and spoofing detection but suffer from lower resolution and higher modality gap, especially when compared to visible or NIR.
• Polarimetry in LWIR has been shown to yield higher recognition accuracy than standard intensity-based thermal imaging, suggesting potential for future high-performance systems.
• Despite advances, CFR remains hampered by insufficient publicly available, large-scale, and diverse datasets, which limits model generalization and benchmarking.