[Paper Review] Thermal to Visible Synthesis of Face Images using Multiple Regions

Benjamin S. Riggan, Nathaniel J. Short+1 more|arXiv (Cornell University)|Mar 20, 2018

Face recognition and analysis20 references5 citations

TL;DR

This paper proposes a multi-region deep learning framework that synthesizes high-quality visible face images from thermal infrared inputs by jointly optimizing global face-level and local facial region (eyes, nose, mouth) representations. By leveraging region-specific feature extraction and regularization, the method achieves state-of-the-art cross-spectrum face verification performance, improving AUC by 5.5% and reducing EER by 3.7% over prior methods, while also enabling accurate landmark detection on synthesized images.

ABSTRACT

Synthesis of visible spectrum faces from thermal facial imagery is a promising approach for heterogeneous face recognition; enabling existing face recognition software trained on visible imagery to be leveraged, and allowing human analysts to verify cross-spectrum matches more effectively. We propose a new synthesis method to enhance the discriminative quality of synthesized visible face imagery by leveraging both global (e.g., entire face) and local regions (e.g., eyes, nose, and mouth). Here, each region provides (1) an independent representation for the corresponding area, and (2) additional regularization terms, which impact the overall quality of synthesized images. We analyze the effects of using multiple regions to synthesize a visible face image from a thermal face. We demonstrate that our approach improves cross-spectrum verification rates over recently published synthesis approaches. Moreover, using our synthesized imagery, we report the results on facial landmark detection-commonly used for image registration-which is a critical part of the face recognition process.

Motivation & Objective

• To address the large modality gap between thermal and visible face imagery, which hinders cross-spectrum face recognition and human verification.
• To enable interoperability with existing visible-spectrum face recognition systems by generating realistic visible-like face images from thermal inputs.
• To improve the discriminative quality of synthesized faces through joint optimization of global and local facial regions.
• To evaluate the utility of synthesized images for downstream tasks such as facial landmark detection.
• To demonstrate that region-based regularization enhances synthesis quality and verification accuracy beyond single-region or global-only approaches.

Proposed method

• A fully convolutional neural network (FCN) extracts features from both a global region of interest (entire face) and multiple local fiducial regions (eyes, nose, mouth).
• Region-specific cross-spectrum mappings are learned to predict corresponding visible features from thermal features.
• Gradient updates from both global and local regions are backpropagated and combined to optimize the synthesis network.
• The method applies complementary regularization terms from each region to improve texture and structural fidelity in the synthesized visible image.
• The framework is trained end-to-end using a perceptual loss that enforces similarity between synthesized and real visible face features.
• Landmark detection is performed on synthesized images using standard visible-spectrum tools (e.g., DLIB) to evaluate practical utility.

Experimental results

Research questions

• RQ1 Can joint optimization over global and local facial regions improve the discriminative quality of thermal-to-visible face synthesis?
• RQ2 Does incorporating region-specific regularization lead to better cross-spectrum verification performance compared to global-only or baseline methods?
• RQ3 Can synthesized visible images from thermal inputs enable accurate landmark detection using standard visible-spectrum tools?
• RQ4 How does the proposed method compare to state-of-the-art synthesis techniques in terms of AUC and EER for cross-spectrum verification?
• RQ5 To what extent does the multi-region approach enhance the usability of synthesized images in existing face recognition pipelines?

Key findings

• The proposed multi-region synthesis method achieves a 5.54% improvement in AUC (85.43%) over the previous state-of-the-art for polarimetric thermal-to-visible synthesis.
• The method reduces the equal error rate (EER) by 3.71% compared to the prior state-of-the-art, achieving 21.46% EER on polarimetric data.
• For conventional thermal inputs, the method achieves 26.25% EER, a 1.11% improvement over the previous best method.
• Landmark detection on synthesized images achieves an average error of 4.95 pixels for conventional thermal inputs and 4.83 pixels for polarimetric thermal inputs.
• The synthesized images enable accurate landmark detection using standard visible-spectrum software (e.g., DLIB), which fails on raw thermal images.
• The framework demonstrates that combining global and local region representations with region-specific regularization significantly enhances synthesis quality and recognition performance.