[Paper Review] Self-adversarial Multi-scale Contrastive Learning for Semantic Segmentation of Thermal Facial Images

Jitesh Joshi, Nadia Bianchi‐Berthouze+1 more|arXiv (Cornell University)|Sep 21, 2022

Face recognition and analysis4 citations

TL;DR

This paper proposes a self-adversarial multi-scale contrastive learning (SAM-CL) framework to improve semantic segmentation of thermal facial images, combining a novel SAM-CL loss with a domain-specific thermal image augmentation (TiAug) module. The method enhances robustness to occlusions and environmental variations, achieving consistent performance gains across multiple networks on limited thermal datasets, including real-world unconstrained settings.

ABSTRACT

Segmentation of thermal facial images is a challenging task. This is because facial features often lack salience due to high-dynamic thermal range scenes and occlusion issues. Limited availability of datasets from unconstrained settings further limits the use of the state-of-the-art segmentation networks, loss functions and learning strategies which have been built and validated for RGB images. To address the challenge, we propose Self-Adversarial Multi-scale Contrastive Learning (SAM-CL) framework as a new training strategy for thermal image segmentation. SAM-CL framework consists of a SAM-CL loss function and a thermal image augmentation (TiAug) module as a domain-specific augmentation technique. We use the Thermal-Face-Database to demonstrate effectiveness of our approach. Experiments conducted on the existing segmentation networks (UNET, Attention-UNET, DeepLabV3 and HRNetv2) evidence the consistent performance gains from the SAM-CL framework. Furthermore, we present a qualitative analysis with UBComfort and DeepBreath datasets to discuss how our proposed methods perform in handling unconstrained situations.

Motivation & Objective

• Address the challenge of semantic segmentation in thermal facial images, which suffer from low feature salience, occlusions, and limited dataset availability.
• Overcome the limitations of existing RGB-based data augmentation and loss functions, which are unsuitable for thermal imaging due to distinct physical properties.
• Develop a training strategy that improves generalization and robustness in unconstrained, real-world thermal imaging scenarios with limited data.
• Design a domain-specific augmentation module (TiAug) that simulates realistic thermal variations, occlusions, and camera noise.
• Introduce a self-adversarial contrastive loss (SAM-CL) that enhances feature discrimination by leveraging synthesized negative samples from class-swapped masks.

Proposed method

• Propose a self-adversarial multi-scale contrastive learning (SAM-CL) loss that encourages feature discrimination by comparing predictions with logits from a class-swapped labelmask.
• Design a thermal image augmentation (TiAug) module that simulates real-world thermal conditions by adjusting ambient temperature, modeling occlusions (e.g., glasses, hair), and incorporating camera noise.
• Apply TiAug to transform controlled laboratory thermal images into realistic, diverse variations, enabling data augmentation without requiring real-world data collection.
• Train segmentation networks (UNet, Attention-UNet, DeepLabV3, HRNetv2) jointly with SAM-CL loss and TiAug to improve robustness and generalization.
• Use multi-scale feature representations to enhance the contrastive learning objective across different receptive field levels.
• Leverage the overlap between predicted logits and synthesized negative samples (from class-swapped masks) to increase loss for incorrect predictions, improving adversarial robustness.

Experimental results

Research questions

• RQ1 Can a self-adversarial contrastive learning framework improve semantic segmentation performance on thermal facial images with limited training data?
• RQ2 How effective is the proposed TiAug module in simulating real-world thermal variations and occlusions without requiring real-world data collection?
• RQ3 Does the SAM-CL loss with TiAug improve generalization across diverse segmentation networks and in unconstrained, real-world scenarios?
• RQ4 To what extent does the framework outperform state-of-the-art methods in handling occlusions such as forehead hair and eyeglasses in thermal images?
• RQ5 Can the SAM-CL framework be generalized to other imaging modalities with limited data by adapting the augmentation and loss components?

Key findings

• The SAM-CL framework consistently improves segmentation performance across multiple architectures (UNet, Attention-UNet, DeepLabV3, HRNetv2) on the Thermal-Face-Database.
• The TiAug module effectively simulates real-world thermal variations, including ambient temperature shifts, occlusions, and camera noise, enhancing model generalization.
• Qualitative analysis shows that models trained with SAM-CL outperform state-of-the-art models in handling forehead hair occlusions, eyeglasses, and low-resolution thermal imaging.
• The model trained with SAM-CL maintains high performance on the DeepBreath dataset (acquired with a mobile thermal camera), demonstrating robustness across different camera specifications.
• The SAM-CL loss with TiAug enables reliable segmentation in unconstrained settings, including images with extreme temperature variations and complex occlusions.
• The framework achieves consistent performance gains without requiring additional real-world data, making it suitable for data-scarce thermal vision applications.