[Paper Review] Semi-Leak: Membership Inference Attacks Against Semi-supervised Learning

Xinlei He, Hongbin Liu+2 more|arXiv (Cornell University)|Jul 25, 2022

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper proposes the first data augmentation-based membership inference attack, Semi-Leak, targeting semi-supervised learning (SSL) models. By exploiting the similarity in model predictions across weak and strong augmentations of the same sample, the attack achieves higher AUC (up to 0.780 on CIFAR10 with 500 labeled samples) than prior methods, revealing that SSL models leak membership information not through overfitting but via overconfidence in predictions on training samples.

ABSTRACT

Semi-supervised learning (SSL) leverages both labeled and unlabeled data to train machine learning (ML) models. State-of-the-art SSL methods can achieve comparable performance to supervised learning by leveraging much fewer labeled data. However, most existing works focus on improving the performance of SSL. In this work, we take a different angle by studying the training data privacy of SSL. Specifically, we propose the first data augmentation-based membership inference attacks against ML models trained by SSL. Given a data sample and the black-box access to a model, the goal of membership inference attack is to determine whether the data sample belongs to the training dataset of the model. Our evaluation shows that the proposed attack can consistently outperform existing membership inference attacks and achieves the best performance against the model trained by SSL. Moreover, we uncover that the reason for membership leakage in SSL is different from the commonly believed one in supervised learning, i.e., overfitting (the gap between training and testing accuracy). We observe that the SSL model is well generalized to the testing data (with almost 0 overfitting) but ''memorizes'' the training data by giving a more confident prediction regardless of its correctness. We also explore early stopping as a countermeasure to prevent membership inference attacks against SSL. The results show that early stopping can mitigate the membership inference attack, but with the cost of model's utility degradation.

Motivation & Objective

• To investigate whether semi-supervised learning (SSL) models leak membership information despite low overfitting.
• To develop a novel data augmentation-based membership inference attack tailored to the unique training dynamics of SSL.
• To analyze the root cause of membership leakage in SSL, contrasting it with overfitting in supervised learning.
• To evaluate defense strategies such as early stopping and their trade-offs between model utility and privacy.

Proposed method

• The attack uses a black-box model to query predictions on weakly and strongly augmented views of a candidate data sample.
• It computes the prediction similarity (e.g., cosine similarity or KL divergence) between the weak and strong augmentations as a membership signal.
• A higher similarity in predictions across augmentations indicates a higher likelihood that the sample is from the training set.
• The attack is evaluated across multiple SSL methods (FixMatch, FlexMatch, UDA) and datasets (SVHN, CIFAR10, CIFAR100).
• The method leverages model confidence (prediction entropy) as a proxy for memorization, even when accuracy is not higher on training samples.
• Defenses such as early stopping, top-k posteriors, model stacking, and DP-SGD are evaluated to mitigate the attack.

Experimental results

Research questions

• RQ1 Does semi-supervised learning introduce unique privacy risks not present in supervised learning?
• RQ2 Can membership inference attacks be more effective by exploiting data augmentation patterns in SSL?
• RQ3 Is overfitting the primary cause of membership leakage in SSL models?
• RQ4 How does model capacity and unlabeled data ratio affect membership inference risk in SSL?
• RQ5 Can early stopping effectively reduce membership inference risk without sacrificing model utility?

Key findings

• The proposed Semi-Leak attack achieves an AUC of 0.780 on FixMatch trained with 500 labeled samples on CIFAR10, outperforming the best baseline (0.722 AUC).
• SSL models exhibit near-zero overfitting (minimal gap between training and test accuracy), contradicting the common belief that overfitting causes membership leakage.
• Despite low overfitting, SSL models still leak membership information by assigning higher prediction confidence to training samples, regardless of label correctness.
• Increasing model capacity (e.g., from WRN28-1 to WRN28-8) increases both model accuracy and membership inference risk, with AUC rising from 0.726 to 0.927.
• Larger unlabeled ratios during training increase membership inference risk, especially for unlabeled samples, with AUC rising from 0.578 (ratio=1) to 0.909 (ratio=16).
• Early stopping provides the best trade-off between model utility and membership privacy, reducing attack performance at the cost of some accuracy degradation.