[Paper Review] Differentially Private CutMix for Split Learning with Vision Transformer

Seungeun Oh, Jihong Park+6 more|arXiv (Cornell University)|Oct 28, 2022

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper proposes DP-CutMixSL, a differentially private split learning framework for Vision Transformers that uses patch-level randomized CutMix to enhance privacy and communication efficiency. By injecting Gaussian noise and replacing random patches in smashed data, it achieves higher accuracy and tighter privacy guarantees than DP-SL and DP-MixSL, with theoretical analysis showing amplified Rényi differential privacy through CutMix operations.

ABSTRACT

Recently, vision transformer (ViT) has started to outpace the conventional CNN in computer vision tasks. Considering privacy-preserving distributed learning with ViT, federated learning (FL) communicates models, which becomes ill-suited due to ViT' s large model size and computing costs. Split learning (SL) detours this by communicating smashed data at a cut-layer, yet suffers from data privacy leakage and large communication costs caused by high similarity between ViT' s smashed data and input data. Motivated by this problem, we propose DP-CutMixSL, a differentially private (DP) SL framework by developing DP patch-level randomized CutMix (DP-CutMix), a novel privacy-preserving inter-client interpolation scheme that replaces randomly selected patches in smashed data. By experiment, we show that DP-CutMixSL not only boosts privacy guarantees and communication efficiency, but also achieves higher accuracy than its Vanilla SL counterpart. Theoretically, we analyze that DP-CutMix amplifies Rényi DP (RDP), which is upper-bounded by its Vanilla Mixup counterpart.

Motivation & Objective

• Address the privacy and communication inefficiency of split learning (SL) with Vision Transformers (ViT), which suffer from high similarity between smashed data and raw inputs.
• Overcome the limitations of standard FL for large ViT models due to high communication and energy costs.
• Develop a privacy-preserving inter-client interpolation mechanism suitable for ViT’s patch-based architecture.
• Theoretically analyze and demonstrate improved differential privacy (DP) guarantees through a novel patch-level randomized CutMix operation.
• Achieve better accuracy-privacy trade-offs compared to existing DP-SL and DP-MixSL baselines.

Proposed method

• Introduce DP-CutMixSL, a differentially private split learning framework using patch-level randomized CutMix for ViT.
• Apply a Gaussian mechanism to inject noise into smashed data, ensuring differential privacy.
• Use a random binary sequence to mask selected patches in the smashed data, creating Cutout smashed data.
• Generate mixing ratios via a Dirichlet multinomial distribution to control patch replacement across clients.
• Combine Cutout data from multiple clients into DP-CutMix smashed data for forward propagation.
• Theoretically bound Rényi differential privacy (RDP) of the method, showing amplification through the CutMix operation.

Experimental results

Research questions

• RQ1 Can patch-level randomized CutMix improve privacy guarantees in split learning for Vision Transformers compared to standard DP-SL?
• RQ2 How does the proposed DP-CutMixSL framework affect the accuracy-privacy trade-off in ViT-based split learning?
• RQ3 Does the use of patch-level interpolation in inter-client data mixing outperform bounding box-based or global mixup methods in ViT?
• RQ4 To what extent does the CutMix operation amplify differential privacy in the context of ViT split learning?
• RQ5 How does the mixing group size influence privacy (ε) and model accuracy in DP-CutMixSL?

Key findings

• DP-CutMixSL achieves the highest top-1 accuracy (73.77%) on CIFAR-10 with ViT-Tiny, outperforming DP-SL and DP-MixSL.
• At a noise variance of 16/255, DP-CutMixSL shows slightly lower accuracy than DP-MixSL, but maintains tighter RDP bounds.
• The method achieves a tighter Rényi differential privacy (RDP) bound than DP-SL, indicating stronger privacy amplification.
• As the mixing group size increases, both privacy (ε) and accuracy decrease, demonstrating the 'hiding in the crowd' effect and information loss trade-off.
• The scalability of CutMixSL is maintained across 2 to 10 clients, with accuracy increasing with more clients, and further improvement when combining with SplitFed’s weight averaging.
• The proposed patch-level CutMix is more effective than bounding box-based Vanilla CutMix in ViT, achieving 71.26% accuracy on PiT-Tiny vs. 58.21% for Vanilla CutMix.