[Paper Review] Fine-Tuning Large Language Models with User-Level Differential Privacy

Zachary Charles, Arun Ganesh+5 more|arXiv (Cornell University)|Jul 10, 2024

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper proposes and evaluates two practical user-level differentially private fine-tuning methods for large language models (LLMs): DP-SGD with example-level sampling (ELS) and user-level sampling (ULS). It introduces a novel, provably tight user-level DP accountant for ELS and finds that ULS generally outperforms ELS in terms of utility, especially under strong privacy constraints or large compute budgets, due to better noise and clipping trade-offs.

ABSTRACT

We investigate practical and scalable algorithms for training large language models (LLMs) with user-level differential privacy (DP) in order to provably safeguard all the examples contributed by each user. We study two variants of DP-SGD with: (1) example-level sampling (ELS) and per-example gradient clipping, and (2) user-level sampling (ULS) and per-user gradient clipping. We derive a novel user-level DP accountant that allows us to compute provably tight privacy guarantees for ELS. Using this, we show that while ELS can outperform ULS in specific settings, ULS generally yields better results when each user has a diverse collection of examples. We validate our findings through experiments in synthetic mean estimation and LLM fine-tuning tasks under fixed compute budgets. We find that ULS is significantly better in settings where either (1) strong privacy guarantees are required, or (2) the compute budget is large. Notably, our focus on LLM-compatible training algorithms allows us to scale to models with hundreds of millions of parameters and datasets with hundreds of thousands of users.

Motivation & Objective

• To develop scalable and practical algorithms for fine-tuning large language models with user-level differential privacy (DP), ensuring all data from each user is protected.
• To address the limitations of example-level DP in settings where users contribute multiple correlated examples, which can lead to user-level membership inference attacks.
• To provide provably tight privacy accounting for user-level DP in the context of LLM training, particularly for the ELS variant.
• To empirically compare the performance of ELS and ULS under fixed compute budgets and varying privacy requirements, identifying optimal configuration strategies.
• To validate the methods on real-world LLM fine-tuning tasks with models up to hundreds of millions of parameters and datasets with hundreds of thousands of users.

Proposed method

• Proposes DP-SGD-ELS: applies DP-SGD with example-level sampling and per-example gradient clipping, then derives user-level privacy guarantees via a novel group privacy analysis.
• Introduces DP-SGD-ULS: applies DP-SGD with user-level sampling and per-user gradient clipping, where gradients are averaged over multiple examples per user before clipping.
• Develops a new user-level DP accountant that computes provably tight privacy bounds for ELS by analyzing the composition of sub-sampled Gaussian mechanisms over grouped examples.
• Employs a synthetic mean estimation task and real LLM fine-tuning tasks (Stack Overflow, CC-News) to compare utility and privacy trade-offs under fixed compute budgets.
• Uses a suite of configuration strategies—including Oracle, Estimate-and-Double, Random, Max Cohort, and Max Group Size—to tune hyperparameters $G_{\text{ULS}}$ and $M$ for ULS.
• Evaluates personalization performance by further fine-tuning the main models on individual user data to assess downstream utility and privacy trade-offs.

Figure 1 : Upper bound on $\varepsilon$ for ELS using our Mixture-of-Gaussians (MoG) accountant and the prior state-of-the-art black-box accounting from [ 23 ] . We set $T=2000$ iterations, sampling probability $p=10^{-2}$ , $\delta=10^{-6}$ , and vary $\sigma_{\text{ELS}}$ and $G_{\text{ELS}}$ . Fo

Experimental results

Research questions

• RQ1 How can user-level differential privacy be effectively and efficiently applied to fine-tune large language models at scale?
• RQ2 What are the relative performance trade-offs between example-level sampling (ELS) and user-level sampling (ULS) in terms of privacy, utility, and computational efficiency?
• RQ3 Can a novel user-level DP accountant provide provably tighter privacy bounds for ELS than generic example-to-user reductions?
• RQ4 Under fixed compute budgets, which method—ELS or ULS—yields better model utility, and under what conditions does each excel?
• RQ5 How does personalization affect the privacy-utility trade-off when applied to models trained with ELS versus ULS?

Key findings

• The proposed user-level DP accountant for ELS provides provably tighter privacy bounds than generic example-to-user reductions, significantly improving privacy-utility trade-offs.
• ULS consistently outperforms ELS in terms of model utility when each user contributes a diverse set of examples, especially under strong privacy constraints ($\varepsilon \leq 16$) or large compute budgets.
• ULS achieves better performance by allowing trade-offs between noise multiplier $\sigma$ and clip norm $C$, whereas ELS is limited to reducing $\sigma$ only, which yields diminishing returns.
• The Estimate-and-Double heuristic for configuring $G_{\text{ULS}}$ and $M$ in ULS performs nearly as well as an oracle strategy across all compute budgets and privacy levels.
• Personalization improves utility for both ELS and ULS, but the relative performance gap between the two methods remains consistent, with ULS maintaining its advantage even after personalization.
• The study demonstrates scalable training of LLMs with user-level DP, achieving effective privacy guarantees on models with hundreds of millions of parameters and datasets with hundreds of thousands of users.

(b) Noise variance of ELS and ULS for varying compute budgets and different relative values of $L_{\text{ELS}}$ and $L_{\text{ULS}}$ .