[Paper Review] Understanding Gradient Clipping in Private SGD: A Geometric Perspective

Xiangyi Chen, Zhiwei Steven Wu+1 more|arXiv (Cornell University)|Jun 27, 2020

Privacy-Preserving Technologies in DataComputer Science17 references61 citations

TL;DR

The paper analyzes how gradient clipping biases SGD and DP-SGD, introduces a symmetricity-based convergence framework, and proposes a perturbation technique to mitigate clipping bias.

ABSTRACT

Deep learning models are increasingly popular in many machine learning applications where the training data may contain sensitive information. To provide formal and rigorous privacy guarantee, many learning systems now incorporate differential privacy by training their models with (differentially) private SGD. A key step in each private SGD update is gradient clipping that shrinks the gradient of an individual example whenever its L2 norm exceeds some threshold. We first demonstrate how gradient clipping can prevent SGD from converging to stationary point. We then provide a theoretical analysis that fully quantifies the clipping bias on convergence with a disparity measure between the gradient distribution and a geometrically symmetric distribution. Our empirical evaluation further suggests that the gradient distributions along the trajectory of private SGD indeed exhibit symmetric structure that favors convergence. Together, our results provide an explanation why private SGD with gradient clipping remains effective in practice despite its potential clipping bias. Finally, we develop a new perturbation-based technique that can provably correct the clipping bias even for instances with highly asymmetric gradient distributions.

Motivation & Objective

• Motivate and quantify how gradient clipping biases convergence in SGD and DP-SGD.
• Develop a symmetricity-based geometric measure to bound clipping bias.
• Extend the analysis to DP-SGD without relying on Lipschitz loss assumptions.
• Empirically validate gradient distribution symmetry during DP-SGD training.
• Propose a perturbation-based technique to reduce clipping bias under skewed gradient distributions.

Proposed method

• Model SGD with gradient clipping and analyze convergence of E[⟨∇f(x_t), g_t⟩] under clipping.
• Introduce a disparity measure via coupling the gradient distribution with a nearby symmetric distribution.
• Establish Theorems 1 and 2 showing positive alignment of clipped gradients under symmetry.
• Extend to DP-SGD with gradient clipping and Gaussian noise, deriving a convergence bound (Theorem 5) involving Wasserstein distance between distributions.
• Propose a pre-clipping perturbation (Gaussian noise added before clipping) to reduce clipping bias (Theorem 6).
• Provide empirical visualizations of gradient symmetry via random projections and cosine histograms.

Experimental results

Research questions

• RQ1 When does gradient clipping preserve convergence in SGD and DP-SGD despite clipping bias?
• RQ2 How does the symmetry of gradient distributions affect the alignment between true and clipped gradients?
• RQ3 Can a perturbation before clipping reduce clipping bias without compromising privacy guarantees?
• RQ4 How does DP-SGD convergence depend on the symmetry measure and Wasserstein distance between gradient distributions?

Key findings

• Clipping bias can prevent convergence in worst-case examples, but symmetry in gradient distributions can ensure positive alignment of clipped gradients with the true gradient.
• A coupling-based disparity measure (resembling Wasserstein distance) bounds clipping bias and explains practical convergence under near-symmetric gradients.
• Empirical results show gradient distributions along DP-SGD trajectories become increasingly symmetric, supporting the theory.
• For DP-SGD, a symmetric gradient distribution recovers the standard O(√d/(nϵ)) convergence rate up to a constant factor; bias is tied to the Wasserstein distance between distributions.
• A perturbation-before-clipping technique provably reduces clipping bias, trading off bias with increased variance.
• Visualization via random projections and cosine histograms indicate approximate symmetry in gradient distributions on MNIST and CIFAR-10.