[Paper Review] FM-ViT: Flexible Modal Vision Transformers for Face Anti-Spoofing

Ajian Liu, Zichang Tan+8 more|arXiv (Cornell University)|May 5, 2023

Biometric Identification and Security4 citations

TL;DR

FM-ViT proposes a pure Vision Transformer-based framework for face anti-spoofing that enables flexible single-modal inference using multi-modal training data. By introducing a Cross-Modal Transformer Block (CMTB) with Multi-headed Mutual-Attention and Fusion-Attention, it enhances modality-specific feature learning and extracts modality-agnostic liveness cues, achieving state-of-the-art performance with lower FLOPs and parameters than multi-modal baselines.

ABSTRACT

The availability of handy multi-modal (i.e., RGB-D) sensors has brought about a surge of face anti-spoofing research. However, the current multi-modal face presentation attack detection (PAD) has two defects: (1) The framework based on multi-modal fusion requires providing modalities consistent with the training input, which seriously limits the deployment scenario. (2) The performance of ConvNet-based model on high fidelity datasets is increasingly limited. In this work, we present a pure transformer-based framework, dubbed the Flexible Modal Vision Transformer (FM-ViT), for face anti-spoofing to flexibly target any single-modal (i.e., RGB) attack scenarios with the help of available multi-modal data. Specifically, FM-ViT retains a specific branch for each modality to capture different modal information and introduces the Cross-Modal Transformer Block (CMTB), which consists of two cascaded attentions named Multi-headed Mutual-Attention (MMA) and Fusion-Attention (MFA) to guide each modal branch to mine potential features from informative patch tokens, and to learn modality-agnostic liveness features by enriching the modal information of own CLS token, respectively. Experiments demonstrate that the single model trained based on FM-ViT can not only flexibly evaluate different modal samples, but also outperforms existing single-modal frameworks by a large margin, and approaches the multi-modal frameworks introduced with smaller FLOPs and model parameters.

Motivation & Objective

• To address the limitation of multi-modal face anti-spoofing models that fail when test modalities differ from training inputs.
• To improve single-modal face anti-spoofing performance using available multi-modal data without requiring consistent modality during inference.
• To develop a pure transformer-based architecture that captures long-range dependencies and global spoofing cues more effectively than CNNs.
• To enable modality-agnostic feature learning through cross-modal attention mechanisms that enrich class token representations.

Proposed method

• FM-ViT employs a multi-branch Vision Transformer architecture, with one branch per input modality (e.g., RGB, Depth, IR), to learn modality-specific features independently.
• The Cross-Modal Transformer Block (CMTB) is inserted after specific stages of the ViT encoder to enable cross-modal feature interaction.
• CMTB consists of two cascaded attention modules: Multi-headed Mutual-Attention (MMA), which identifies informative patch tokens from one modality using attention maps from another, and Fusion-Attention (MFA), which enriches the CLS token with cross-modal information.
• MMA uses a learnable threshold function Γλ(⋅) to retain the top 50% of attention mass, ensuring selective feature mining across modalities.
• MFA performs cross-attention between the CLS token of one modality and patch tokens from other modalities, enabling the model to learn modality-invariant liveness representations.
• The framework is trained on multi-modal data but can be deployed for inference on any single modality, enabling flexible modal testing.

Experimental results

Research questions

• RQ1 Can a single model trained on multi-modal data generalize effectively to single-modal testing scenarios without requiring consistent modality during inference?
• RQ2 How can cross-modal attention mechanisms improve the detection of subtle spoofing traces in single-modal inputs?
• RQ3 What is the optimal balance between modality-specific feature learning and modality-agnostic representation learning in a pure transformer-based FAS framework?
• RQ4 To what extent can a lightweight transformer module enhance performance while reducing FLOPs and model parameters compared to multi-modal fusion baselines?

Key findings

• The FM-ViT(S) variant achieves an ACER of 2.87%/2.32%/2.13% on the R/D/I protocols of the OULU-NPU dataset, outperforming the baseline ViT(S) by 1.45 percentage points on average.
• FM-ViT(S) reduces FLOPs by 90% and parameters by 91% compared to the full multi-modal FM-ViT(B), while maintaining superior performance.
• The ablation study confirms that λ=0.5 in the MMA thresholding function yields optimal performance, balancing informative token selection and feature diversity.
• Visualization results show that MMA effectively redirects attention to spoofing-relevant regions—such as paper edges or glasses—across modalities, even when the target modality misses these cues.
• MFA enhances feature relevance maps across modalities, expanding the response areas of salient features and improving robustness to subtle spoofing artifacts.
• The model achieves state-of-the-art performance on multiple benchmarks, including OULU-NPU, WMCA, and MmFa, with strong zero-shot generalization across unseen protocols.