[Paper Review] Protecting Split Learning by Potential Energy Loss

Fei Zheng, Chaochao Chen+6 more|arXiv (Cornell University)|Oct 18, 2022

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper proposes Potential Energy Loss (PELoss), a novel regularization method that increases the generalization error of attackers performing label leakage attacks on split learning by pushing same-class outputs toward the decision boundary. By modeling inter-class output repulsion akin to electrostatic repulsion, PELoss makes the bottom model's outputs more complex and harder to fine-tune with few labeled samples, significantly reducing attacker accuracy while maintaining acceptable model performance.

ABSTRACT

As a practical privacy-preserving learning method, split learning has drawn much attention in academia and industry. However, its security is constantly being questioned since the intermediate results are shared during training and inference. In this paper, we focus on the privacy leakage from the forward embeddings of split learning. Specifically, since the forward embeddings contain too much information about the label, the attacker can either use a few labeled samples to fine-tune the top model or perform unsupervised attacks such as clustering to infer the true labels from the forward embeddings. To prevent such kind of privacy leakage, we propose the potential energy loss to make the forward embeddings become more 'complicated', by pushing embeddings of the same class towards the decision boundary. Therefore, it is hard for the attacker to learn from the forward embeddings. Experiment results show that our method significantly lowers the performance of both fine-tuning attacks and clustering attacks.

Motivation & Objective

• Address the privacy leakage risk in split learning where an attacker can reconstruct labels and fine-tune the bottom model using only a few labeled samples.
• Identify that the core issue lies in the compact, clustered distribution of same-class outputs in standard split learning, which enables effective fine-tuning by adversaries.
• Reframe the privacy protection problem as a generalization error maximization task: make it hard for attackers to generalize from few labeled samples.
• Propose a physically inspired loss function—Potential Energy Loss—that redistributes same-class outputs toward the decision boundary to increase attacker uncertainty.
• Demonstrate that PELoss achieves stronger privacy protection than existing methods like distance correlation loss (DcorLoss), with better stability and performance trade-offs.

Proposed method

• Introduce a potential energy loss term that models repulsive forces between all pairs of same-class outputs in the bottom model's feature space.
• The loss function is defined as the sum of inverse pairwise distances between same-class outputs, scaled by a hyperparameter α, encouraging outputs to spread out toward the decision boundary.
• Integrate PELoss as a regularizer during split learning training, modifying the loss landscape to make same-class outputs less clustered and more dispersed.
• Train the bottom model with PELoss while maintaining standard split learning communication and optimization protocols.
• Use a hyperparameter α to control the strength of repulsion: higher α increases the penalty for compact same-class clusters.
• Apply early stopping and model saving strategies to ensure convergence and robustness, with evaluation on both test accuracy and attacker fine-tuning performance.

Figure 1: Label leakage attack of split learning.

Experimental results

Research questions

• RQ1 Can we reduce the success of label leakage attacks in split learning by modifying the output distribution of the bottom model?
• RQ2 Does pushing same-class outputs toward the decision boundary increase the generalization error of an attacker fine-tuning the bottom model with only a few labeled samples?
• RQ3 How does the proposed Potential Energy Loss compare to existing methods like distance correlation loss in terms of privacy protection and model performance?
• RQ4 Is the proposed method stable across different datasets and hyperparameter settings, especially as the repulsion strength increases?
• RQ5 To what extent does PELoss affect the convergence speed and final accuracy of the main learning task?

Key findings

• PELoss significantly reduces the attacker’s fine-tuning accuracy across all datasets (MNIST, Fashion-MNIST, CIFAR-10), with the attacker’s accuracy dropping below the baseline of training from scratch using the same leaked labels when α ≥ 8.
• On average, PELoss achieves lower attacker accuracy than DcorLoss at the same test accuracy level, demonstrating superior privacy protection capability.
• The PELoss curve is smoother and more responsive to α than DcorLoss, enabling better and more predictable trade-offs between privacy and performance.
• Training with PELoss remains stable even at high α values (e.g., α = 16), whereas DcorLoss often causes training divergence under similar conditions.
• The convergence speed is only slightly slower with PELoss, with minimal impact on training efficiency, making it practical for real-world deployment.
• t-SNE visualization confirms that PELoss effectively spreads out same-class outputs, causing clusters to overlap and become less separable—especially noticeable in CIFAR-10 due to the model’s higher capacity.

Figure 2: A small error on the decision region caused by small sample size (left) leads to a large generalization error (right).