[Paper Review] FaceMAE: Privacy-Preserving Face Recognition via Masked Autoencoders

Kai Wang, Bo Zhao+6 more|arXiv (Cornell University)|May 23, 2022

Face recognition and analysis6 citations

TL;DR

FaceMAE proposes a privacy-preserving face recognition framework using masked autoencoders to reconstruct face images from masked inputs while preserving recognition utility. By replacing standard MSE loss with instance relation matching (IRM), FaceMAE minimizes distribution gaps between real and reconstructed faces, reducing membership privacy leakage by ~20% and cutting recognition error by at least 50% on LFW, CFP-FP, and AgeDB compared to state-of-the-art methods.

ABSTRACT

Face recognition, as one of the most successful applications in artificial intelligence, has been widely used in security, administration, advertising, and healthcare. However, the privacy issues of public face datasets have attracted increasing attention in recent years. Previous works simply mask most areas of faces or synthesize samples using generative models to construct privacy-preserving face datasets, which overlooks the trade-off between privacy protection and data utility. In this paper, we propose a novel framework FaceMAE, where the face privacy and recognition performance are considered simultaneously. Firstly, randomly masked face images are used to train the reconstruction module in FaceMAE. We tailor the instance relation matching (IRM) module to minimize the distribution gap between real faces and FaceMAE reconstructed ones. During the deployment phase, we use trained FaceMAE to reconstruct images from masked faces of unseen identities without extra training. The risk of privacy leakage is measured based on face retrieval between reconstructed and original datasets. Experiments prove that the identities of reconstructed images are difficult to be retrieved. We also perform sufficient privacy-preserving face recognition on several public face datasets (i.e. CASIA-WebFace and WebFace260M). Compared to previous state of the arts, FaceMAE consistently extbf{reduces at least 50\% error rate} on LFW, CFP-FP and AgeDB.

Motivation & Objective

• Address the growing concern over membership privacy leakage in public face datasets used for training deep face recognition models.
• Overcome limitations of prior methods—such as blurring, masking, or GAN-based synthesis—that degrade data utility or fail to generalize to unseen identities.
• Simultaneously preserve privacy and maintain high recognition performance by reconstructing semantically meaningful, identity-preserving faces from masked inputs.
• Develop a framework that enables deployment on unseen identities without retraining, ensuring scalability and practicality.
• Minimize the risk of membership inference attacks by making reconstructed faces indistinguishable from originals in retrieval tasks.

Proposed method

• Train a masked autoencoder (MAE) on randomly masked face images to reconstruct the original face distribution.
• Replace the standard MSE loss with an instance relation matching (IRM) module to align the feature relation graphs of real and reconstructed faces.
• Use IRM to minimize the distribution gap between real and reconstructed face features, improving recognition utility.
• During deployment, apply the trained FaceMAE to reconstruct masked faces of unseen identities without additional fine-tuning.
• Measure privacy leakage via face retrieval accuracy between original and reconstructed datasets to evaluate membership privacy protection.
• Optimize the trade-off between privacy and performance using a hyperparameter β that controls the weight of IRM loss.

Experimental results

Research questions

• RQ1 Can masked autoencoders with instance relation matching reconstruct face images that preserve recognition utility while reducing privacy leakage?
• RQ2 How does FaceMAE compare to traditional masking and GAN-based methods in terms of face recognition accuracy and privacy protection?
• RQ3 To what extent does the IRM module reduce the distribution gap between real and reconstructed face features?
• RQ4 How robust is FaceMAE to different mask patterns and locations (e.g., eyes, mouth)?
• RQ5 What is the impact of the IRM loss weighting hyperparameter β on the balance between privacy and recognition performance?

Key findings

• FaceMAE reduces the face recognition error rate by at least 50% on LFW, CFP-FP, and AgeDB compared to state-of-the-art methods.
• The privacy leakage risk, measured by retrieval accuracy between original and reconstructed datasets, decreases by approximately 20%.
• Reconstruction with FaceMAE achieves significantly better feature distribution alignment with real faces than standard MAE or GAN-based methods.
• The method maintains strong performance even when masking critical facial regions like eyes and mouth, though performance degrades slightly compared to full-face masking.
• Increasing the IRM loss weight β beyond 1.0 leads to a performance drop, indicating an optimal trade-off point around β = 1.0.
• The retrieval performance of 'original to FaceMAE' is much lower than 'original to original', confirming effective privacy protection, and is also lower than 'original to MAE', showing FaceMAE's superiority.