[Paper Review] Unsupervised Enhancement of Soft-biometric Privacy with Negative Face Recognition

Philipp Terhörst, Marco Huber+3 more|arXiv (Cornell University)|Feb 21, 2020

Biometric Identification and Security35 references18 citations

TL;DR

This paper proposes Negative Face Recognition (NFR), an unsupervised method that enhances soft-biometric privacy by storing only negative, complementary facial feature templates instead of real ones. By comparing probe templates against these negative references, the approach suppresses privacy-sensitive attributes like gender, age, and race—achieving up to 36% suppression in uncontrolled conditions while maintaining full recognition performance, outperforming prior supervised methods in both privacy protection and accuracy.

ABSTRACT

Current research on soft-biometrics showed that privacy-sensitive information can be deduced from biometric templates of an individual. Since for many applications, these templates are expected to be used for recognition purposes only, this raises major privacy issues. Previous works focused on supervised privacy-enhancing solutions that require privacy-sensitive information about individuals and limit their application to the suppression of single and pre-defined attributes. Consequently, they do not take into account attributes that are not considered in the training. In this work, we present Negative Face Recognition (NFR), a novel face recognition approach that enhances the soft-biometric privacy on the template-level by representing face templates in a complementary (negative) domain. While ordinary templates characterize facial properties of an individual, negative templates describe facial properties that does not exist for this individual. This suppresses privacy-sensitive information from stored templates. Experiments are conducted on two publicly available datasets captured under controlled and uncontrolled scenarios on three privacy-sensitive attributes. The experiments demonstrate that our proposed approach reaches higher suppression rates than previous work, while maintaining higher recognition performances as well. Unlike previous works, our approach does not require privacy-sensitive labels and offers a more comprehensive privacy-protection not limited to pre-defined attributes.

Motivation & Objective

• To address privacy leaks from biometric templates that reveal soft-biometric attributes like gender, age, and ethnicity.
• To overcome limitations of supervised methods that require privacy-labeled training data and are restricted to pre-defined attributes.
• To design a privacy-preserving face recognition system resilient against adaptive function creep attacks.
• To maintain high face recognition performance while suppressing unintended attribute inference.
• To provide a template-level, unsupervised solution that is non-invertible, revocable, and non-linkable by design.

Proposed method

• The method introduces negative face templates that represent facial features not present in the individual, using a binarization strategy based on k-means clustering of deep features.
• Face templates are encoded into binary codes using a learned quantization function, with k=3 or k=4 bins to balance suppression and performance.
• Recognition is performed by comparing a positive probe template against negative reference templates, leveraging a theoretical score distribution model to ensure reliability.
• A theoretical framework models the expected score distribution in the negative-positive comparison space, validated empirically on two datasets.
• The approach uses a complementary representation: while positive templates describe actual facial properties, negative templates encode only non-existent, random complementary features.
• The system is trained without any privacy-sensitive labels, making it unsupervised and scalable to unseen attributes.

Experimental results

Research questions

• RQ1 Can an unsupervised method suppress privacy-sensitive attributes from face templates without requiring labeled privacy data?
• RQ2 How effective is negative face recognition in suppressing gender, age, and race attributes under both controlled and uncontrolled image conditions?
• RQ3 Can the method maintain high face recognition accuracy while achieving strong attribute suppression?
• RQ4 How does the system perform against an adaptive function creep attacker who knows the privacy mechanism?
• RQ5 What is the theoretical and empirical relationship between template bin size (k) and the trade-off between recognition performance and attribute suppression?

Key findings

• The proposed NFR method achieves up to 36% suppression of privacy-sensitive attributes in uncontrolled scenarios, significantly outperforming prior supervised methods.
• In uncontrolled conditions, NFR maintains full recognition performance (100% verification accuracy) while suppressing attributes more effectively than previous approaches.
• The method achieves 2-4 times higher suppression rates than prior work across multiple attributes and attack models, even when attackers adapt to the system.
• The theoretical score prediction model closely matches empirical score distributions on both the ColorFeret and Adience datasets, validating the method’s theoretical foundation.
• The optimal bin size is k=3, balancing high attribute suppression and low verification error, with k=4 showing higher variability and performance degradation.
• The approach is unsupervised and does not require privacy-labeled data, enabling comprehensive protection beyond pre-defined attributes.