[Paper Review] Rethinking Vision Transformer and Masked Autoencoder in Multimodal Face Anti-Spoofing

Zitong Yu, Rizhao Cai+4 more|arXiv (Cornell University)|Feb 11, 2023

Biometric Identification and Security4 citations

TL;DR

This paper rethinks Vision Transformers (ViT) and Masked Autoencoders (MAE) for multimodal face anti-spoofing (FAS) by introducing modality-aware inputs, a modality-asymmetric masked autoencoder (M²A²E) for self-supervised pre-training, and an adaptive multimodal adapter (AMA) for efficient finetuning. The proposed methods achieve state-of-the-art performance across unimodal and multimodal FAS benchmarks, demonstrating superior generalization and robustness on RGB, IR, and Depth modalities.

ABSTRACT

Recently, vision transformer (ViT) based multimodal learning methods have been proposed to improve the robustness of face anti-spoofing (FAS) systems. However, there are still no works to explore the fundamental natures ( extit{e.g.}, modality-aware inputs, suitable multimodal pre-training, and efficient finetuning) in vanilla ViT for multimodal FAS. In this paper, we investigate three key factors (i.e., inputs, pre-training, and finetuning) in ViT for multimodal FAS with RGB, Infrared (IR), and Depth. First, in terms of the ViT inputs, we find that leveraging local feature descriptors benefits the ViT on IR modality but not RGB or Depth modalities. Second, in observation of the inefficiency on direct finetuning the whole or partial ViT, we design an adaptive multimodal adapter (AMA), which can efficiently aggregate local multimodal features while freezing majority of ViT parameters. Finally, in consideration of the task (FAS vs. generic object classification) and modality (multimodal vs. unimodal) gaps, ImageNet pre-trained models might be sub-optimal for the multimodal FAS task. To bridge these gaps, we propose the modality-asymmetric masked autoencoder (M$^{2}$A$^{2}$E) for multimodal FAS self-supervised pre-training without costly annotated labels. Compared with the previous modality-symmetric autoencoder, the proposed M$^{2}$A$^{2}$E is able to learn more intrinsic task-aware representation and compatible with modality-agnostic (e.g., unimodal, bimodal, and trimodal) downstream settings. Extensive experiments with both unimodal (RGB, Depth, IR) and multimodal (RGB+Depth, RGB+IR, Depth+IR, RGB+Depth+IR) settings conducted on multimodal FAS benchmarks demonstrate the superior performance of the proposed methods. We hope these findings and solutions can facilitate the future research for ViT-based multimodal FAS.

Motivation & Objective

• To investigate the fundamental challenges in applying Vision Transformers (ViT) to multimodal face anti-spoofing (FAS), including modality-aware inputs, suitable pre-training, and efficient finetuning.
• To address the limitations of vanilla ViT in handling modality-specific representations, especially for infrared (IR) modality, by incorporating local feature descriptors.
• To overcome the sub-optimality of ImageNet pre-training for FAS by designing a modality-asymmetric masked autoencoder (M²A²E) for task-specific self-supervised pre-training.
• To enable efficient and effective finetuning of ViT in multimodal FAS through an adaptive multimodal adapter (AMA) that freezes most ViT parameters while aggregating cross-modal features.

Proposed method

• Introduces modality-specific input processing: using local feature descriptors (e.g., HOG, PLGF) for IR modality while using raw inputs for RGB and Depth, to enhance local and illumination-invariant representations.
• Proposes a modality-asymmetric masked autoencoder (M²A²E) that masks and reconstructs inputs asymmetrically across modalities, enabling task-aware, generalized representation learning without labeled data.
• Designs an adaptive multimodal adapter (AMA) that efficiently fuses local multimodal features while freezing the majority of ViT parameters, enabling parameter-efficient finetuning.
• Employs a multi-stage training strategy: pre-train M²A²E on unlabeled data, then finetune with AMA on downstream FAS tasks, leveraging both self-supervised and supervised signals.
• Uses a hierarchical masking strategy in M²A²E, where different modalities are masked at varying ratios and reconstructed with modality-specific decoders to preserve task-relevant cues.
• Validates the method across multiple benchmarks (WMCA, CeFA) under both seen and unseen attack protocols, evaluating unimodal and multimodal settings (e.g., RGB+IR+Depth).

Experimental results

Research questions

• RQ1 Does incorporating local feature descriptors into ViT inputs improve performance on the infrared (IR) modality in multimodal FAS?
• RQ2 Can a modality-asymmetric masked autoencoder (M²A²E) outperform symmetric MAE and ImageNet pre-training for self-supervised pre-training in multimodal FAS?
• RQ3 How effective is the adaptive multimodal adapter (AMA) in enabling efficient finetuning of ViT while maintaining high performance across unimodal and multimodal settings?
• RQ4 Does the combination of modality-aware inputs, M²A²E pre-training, and AMA finetuning lead to superior generalization on unseen attacks and deployment conditions?

Key findings

• Using local feature descriptors (e.g., HOG, PLGF) for IR modality inputs significantly improves ViT performance, especially in bimodal and trimodal settings, due to enhanced local and illumination-invariant feature representation.
• The proposed modality-asymmetric masked autoencoder (M²A²E) outperforms both symmetric multimodal MAE and ImageNet pre-training across all unimodal and multimodal FAS settings, demonstrating superior downstream generalization.
• The adaptive multimodal adapter (AMA) enables efficient finetuning with minimal parameter updates, achieving competitive performance while freezing most ViT layers, thus reducing overfitting risks on small FAS datasets.
• On the WMCA benchmark under the 'seen' protocol, the full pipeline (M²A²E pre-training + AMA finetuning + modality-aware inputs) achieves a 100% EER on RGB+IR+Depth, outperforming prior methods.
• Visualization of M²A²E reconstructions confirms that the model preserves critical live/spoof cues (e.g., facial depth, texture) even from masked inputs, validating its ability to learn discriminative, task-aware representations.
• Ablation studies show that finetuning only the last transformer block and classification head yields the best trade-off between performance and efficiency, outperforming full finetuning due to reduced overfitting.