[Paper Review] Practical Blind Membership Inference Attack via Differential Comparisons

Bo Hui, Yuchen Yang+4 more|arXiv (Cornell University)|Jan 5, 2021

Privacy-Preserving Technologies in Data48 references4 citations

TL;DR

This paper proposes BlindMI, a novel blackbox membership inference attack that bypasses the need for shadow models by using differential comparison to infer membership status through iterative sample movement between datasets. It achieves state-of-the-art performance, improving F1-score by nearly 20% on datasets like Purchase-50 and Birds-200, and remains effective against advanced defenses.

ABSTRACT

Membership inference (MI) attacks affect user privacy by inferring whether given data samples have been used to train a target learning model, e.g., a deep neural network. There are two types of MI attacks in the literature, i.e., these with and without shadow models. The success of the former heavily depends on the quality of the shadow model, i.e., the transferability between the shadow and the target; the latter, given only blackbox probing access to the target model, cannot make an effective inference of unknowns, compared with MI attacks using shadow models, due to the insufficient number of qualified samples labeled with ground truth membership information. In this paper, we propose an MI attack, called BlindMI, which probes the target model and extracts membership semantics via a novel approach, called differential comparison. The high-level idea is that BlindMI first generates a dataset with nonmembers via transforming existing samples into new samples, and then differentially moves samples from a target dataset to the generated, non-member set in an iterative manner. If the differential move of a sample increases the set distance, BlindMI considers the sample as non-member and vice versa. BlindMI was evaluated by comparing it with state-of-the-art MI attack algorithms. Our evaluation shows that BlindMI improves F1-score by nearly 20% when compared to state-of-the-art on some datasets, such as Purchase-50 and Birds-200, in the blind setting where the adversary does not know the target model's architecture and the target dataset's ground truth labels. We also show that BlindMI can defeat state-of-the-art defenses.

Motivation & Objective

• To address the limitation of existing membership inference attacks that rely on shadow models with poor transferability to the target model.
• To enable effective membership inference in a blackbox setting where the adversary has no access to the target model’s architecture or ground-truth labels.
• To develop a method that extracts complex membership semantics from probing results alone, without requiring labeled member/non-member data.
• To evaluate the robustness of BlindMI against state-of-the-art defenses in membership inference.
• To demonstrate that blackbox attacks can achieve performance comparable to whitebox attacks through intelligent probing and comparison strategies.

Proposed method

• BlindMI generates a non-member dataset by transforming existing samples into new, semantically plausible samples assumed to be non-members with high confidence.
• It uses a differential comparison mechanism: iteratively moving a sample from a target dataset to the non-member set and measuring the change in set distance in the feature space.
• If the move decreases the distance between the two sets, the sample is classified as a member; if it increases the distance, the sample is classified as a non-member.
• The method leverages the geometric intuition that moving a member toward a non-member set reduces the overall distance, while moving a non-member increases it.
• Three variants are implemented: BlindMI-1class (using one-class SVM for non-member detection), BlindMI-diff-w/ (with generated non-members), and BlindMI-diff-w/o (with rough member/non-member separation).
• The approach avoids reliance on pre-defined thresholds or ground-truth labels, instead learning the decision boundary implicitly through dynamic set comparisons.

Experimental results

Research questions

• RQ1 Can membership inference be effectively performed in a blackbox setting without access to ground-truth labels or shadow models?
• RQ2 Can differential comparison of dataset sets in feature space reveal meaningful membership semantics without explicit model knowledge?
• RQ3 How does BlindMI compare in performance to state-of-the-art membership inference attacks in terms of F1-score and robustness?
• RQ4 To what extent can BlindMI bypass or defeat existing defenses such as regularization, adversarial training, and differential privacy?
• RQ5 Does the proposed method outperform threshold-based or one-class learning approaches in capturing complex decision boundaries in high-dimensional space?

Key findings

• BlindMI improves the F1-score by nearly 20% compared to state-of-the-art attacks on datasets such as Purchase-50 and Birds-200 in the blind setting.
• The method achieves strong performance without requiring knowledge of the target model’s architecture or the ground-truth labels of the training data.
• BlindMI-diff-w/ and BlindMI-diff-w/o variants outperform both threshold-based and one-class SVM baselines, demonstrating the effectiveness of differential comparison.
• BlindMI remains effective against state-of-the-art defenses, including regularization, adversarial training, and differential privacy, achieving an F1-score above 60% in many cases.
• The attack is robust across diverse DNN architectures and datasets, confirming its generalizability and practicality in real-world blackbox scenarios.
• The results validate that blackbox membership inference can be as powerful as whitebox attacks when the probing strategy captures complex geometric relationships in the feature space.