[Paper Review] On the Efficacy of Differentially Private Few-shot Image Classification

Marlon Tobaben, Aliaksandra Shysheya+6 more|arXiv (Cornell University)|Feb 2, 2023

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper investigates differentially private (DP) few-shot image classification, proposing a comprehensive evaluation of DP-SGD with varying shots per class, privacy levels, model architectures, and parameter-efficient fine-tuning strategies. It demonstrates that FiLM adapters enable DP performance comparable to full fine-tuning while improving communication efficiency, achieving state-of-the-art accuracy (51.9% macro F1) on the VTAB-1k benchmark under DP.

ABSTRACT

There has been significant recent progress in training differentially private (DP) models which achieve accuracy that approaches the best non-private models. These DP models are typically pretrained on large public datasets and then fine-tuned on private downstream datasets that are relatively large and similar in distribution to the pretraining data. However, in many applications including personalization and federated learning, it is crucial to perform well (i) in the few-shot setting, as obtaining large amounts of labeled data may be problematic; and (ii) on datasets from a wide variety of domains for use in various specialist settings. To understand under which conditions few-shot DP can be effective, we perform an exhaustive set of experiments that reveals how the accuracy and vulnerability to attack of few-shot DP image classification models are affected as the number of shots per class, privacy level, model architecture, downstream dataset, and subset of learnable parameters in the model vary. We show that to achieve DP accuracy on par with non-private models, the shots per class must be increased as the privacy level increases. We also show that learning parameter-efficient FiLM adapters under DP is competitive with learning just the final classifier layer or learning all of the network parameters. Finally, we evaluate DP federated learning systems and establish state-of-the-art performance on the challenging FLAIR benchmark.

Motivation & Objective

• To investigate the conditions under which differentially private (DP) few-shot image classification is effective, especially in low-data and out-of-domain settings.
• To evaluate the trade-offs between privacy level (ε), number of shots per class, and model accuracy in DP fine-tuning.
• To assess the efficacy of parameter-efficient fine-tuning methods—such as FiLM adapters, final-layer-only, and full fine-tuning—under DP in few-shot regimes.
• To extend findings to federated learning, evaluating DP performance on the challenging FLAIR benchmark.
• To provide a systematic comparison of privacy accounting methods (RDP and PRV) for DP-SGD in few-shot settings.

Proposed method

• Employs DP-SGD with gradient clipping and noise injection to train models under (ε,δ)-differential privacy, using the RDP and PRV accountants for privacy accounting.
• Performs extensive ablation studies varying shots per class (1–500), privacy budgets (ε ∈ {1,2,4,8}), downstream datasets (CIFAR-10, CIFAR-100, SVHN, VTAB-1k), and model architectures (ResNet, ViT).
• Evaluates three fine-tuning strategies: (1) only the final classifier layer, (2) all model parameters, and (3) parameter-efficient FiLM adapters applied to intermediate layers.
• Applies the same training protocol to both centralized and federated learning settings, with hyperparameter search over client and server learning rates and adaptive clipping thresholds.
• Uses the VTAB-1k benchmark to evaluate out-of-domain transfer performance and communication efficiency in federated DP learning.
• Recomputes privacy guarantees using both RDP and PRV accountants to ensure robustness and accuracy of ε estimates under varying δ (e.g., δ = 1e−5).

Figure 1 : Classification accuracy as a function of shots and $\epsilon$ for CIFAR-10, CIFAR-100 and SVHN. Backbone is VIT-B and the best performing configuration out of All , FiLM and Head is used for each combination of $\epsilon$ and $S$ , with $\delta=1/|\mathcal{D}|$ . The accuracy is reported

Experimental results

Research questions

• RQ1 How does the number of shots per class affect the accuracy of differentially private few-shot image classification under varying privacy budgets (ε)?
• RQ2 How does the choice of fine-tuning strategy—final layer, full network, or FiLM adapters—affect DP accuracy and communication efficiency in few-shot settings?
• RQ3 To what extent does out-of-domain transfer (e.g., from ImageNet to VTAB-1k) degrade DP model performance, and how can it be mitigated?
• RQ4 Do trends observed in centralized DP training generalize to federated DP learning, particularly in terms of privacy-accuracy trade-offs?
• RQ5 How do different privacy accounting methods (RDP vs. PRV) affect the computed ε values and their comparability across experiments?

Key findings

• To match non-private model accuracy under DP, the number of shots per class must be significantly increased as the privacy budget (ε) decreases, with a clear data-privacy trade-off.
• Fine-tuning FiLM adapters in addition to the final classifier layer achieves state-of-the-art performance on the VTAB-1k benchmark under DP, reaching 51.9% macro F1, a substantial improvement over the prior SOTA of 44.3%.
• FiLM adapter fine-tuning under DP outperforms both full fine-tuning and final-layer-only fine-tuning in terms of accuracy and communication efficiency, especially in low-shot and out-of-domain settings.
• The accuracy of DP models is strongly correlated with the difficulty of the transfer learning task, with more complex or out-of-domain datasets requiring more data or better parameterization to maintain performance.
• Privacy accounting using the PRV accountant yields slightly lower ε estimates than RDP, improving privacy guarantees while maintaining comparable model performance.
• In federated learning, the proposed DP-FedAvg setup achieves state-of-the-art performance on the FLAIR benchmark, demonstrating the viability of DP in real-world, decentralized settings with limited data per client.