[Paper Review] Two-stream Convolutional Networks for Multi-frame Face Anti-spoofing

Zhuoyi Zhang, Cheng Jiang+3 more|arXiv (Cornell University)|Aug 9, 2021

Biometric Identification and Security34 references4 citations

TL;DR

This paper proposes a lightweight two-stream convolutional network for multi-frame face anti-spoofing that leverages both multi-frame RGB inputs and RGB difference images to capture complementary spatial-temporal features. By integrating feature pyramid modules with opposing fusion directions and pyramid pooling modules, the model achieves state-of-the-art performance on multiple benchmarks with significantly fewer parameters, outperforming existing methods in accuracy and efficiency.

ABSTRACT

Face anti-spoofing is an important task to protect the security of face recognition. Most of previous work either struggle to capture discriminative and generalizable feature or rely on auxiliary information which is unavailable for most of industrial product. Inspired by the video classification work, we propose an efficient two-stream model to capture the key differences between live and spoof faces, which takes multi-frames and RGB difference as input respectively. Feature pyramid modules with two opposite fusion directions and pyramid pooling modules are applied to enhance feature representation. We evaluate the proposed method on the datasets of Siw, Oulu-NPU, CASIA-MFSD and Replay-Attack. The results show that our model achieves the state-of-the-art results on most of datasets' protocol with much less parameter size.

Motivation & Objective

• To address the challenge of distinguishing live faces from spoof attacks (e.g., print, replay) in real-world face recognition systems.
• To develop a lightweight, efficient model that avoids reliance on auxiliary data like depth or infrared, which are often unavailable in industrial applications.
• To enhance feature representation by capturing both fine-grained motion cues and robust spatial patterns across multiple video frames.
• To reduce overfitting caused by redundant facial appearance features in RGB images through the use of RGB difference inputs and attention mechanisms.

Proposed method

• The model employs a two-stream architecture: one stream processes raw multi-frame RGB inputs using a CNN-LSTM backbone with an opposite-direction feature pyramid module (FPM) and pyramid pooling module (PPM).
• The second stream processes stacked RGB difference images—computed as pixel-wise differences between consecutive frames—to emphasize motion and temporal dynamics while reducing overfitting to facial appearance.
• Feature pyramid modules with opposing fusion directions enhance multi-scale feature learning, improving representation of both subtle motion and structural artifacts.
• Pyramid pooling modules aggregate features across spatial scales, enhancing robustness to variations in face size and pose.
• A branch supervision strategy applies spatial attention masks from the RGB difference branch to guide feature learning in the multi-frame branch, though experiments show limited benefit from this fusion.
• The final prediction is made by concatenating and classifying the score outputs from both branches, combining their complementary strengths.

Experimental results

Research questions

• RQ1 Can a two-stream CNN architecture effectively exploit both multi-frame RGB and RGB difference inputs to improve face anti-spoofing performance?
• RQ2 How can feature pyramid modules with opposing fusion directions enhance spatial-temporal feature learning in low-resource settings?
• RQ3 To what extent does using RGB difference images reduce overfitting to facial appearance while preserving spoofing-relevant motion cues?
• RQ4 Does the integration of pyramid pooling and feature pyramid modules lead to better generalization across diverse anti-spoofing protocols?
• RQ5 Can a lightweight model achieve state-of-the-art performance without relying on auxiliary data such as depth or infrared?

Key findings

• The proposed two-stream model achieves the state-of-the-art performance on most protocols of the SiW, Oulu-NPU, CASIA-MFSD, and Replay-Attack datasets, with an ACER of 1.2% on the SiW dataset under cross-testing.
• The model achieves a HTER of 8.0% on the SiW dataset, significantly outperforming the baseline single-stream model (HTER: 11.0%) and other ablation variants.
• The use of RGB difference images as input to one stream effectively reduces overfitting and enhances detection of subtle motion artifacts, contributing to improved generalization.
• Among ablation studies, the style transfer and denoising methods improved performance under cross-testing, but at the cost of increased parameter size, indicating a trade-off between accuracy and efficiency.
• The branch supervision strategy, intended to improve feature learning via cross-branch attention, did not yield performance gains, suggesting that independent learning of complementary features is more effective.
• The model’s compact design—using a modified MobileNetV3-Small backbone—achieves high accuracy with significantly fewer parameters than prior SOTA methods, making it suitable for industrial deployment.