[Paper Review] KNG: The K-Norm Gradient Mechanism

Matthew Reimherr, Jordan Awan|arXiv (Cornell University)|May 23, 2019

Privacy-Preserving Technologies in Data31 references4 citations

TL;DR

This paper introduces the K-Norm Gradient Mechanism (KNG), a novel differentially private algorithm that enhances utility by using the gradient of an objective function and a flexible K-norm to weight candidate solutions. Unlike the exponential mechanism, KNG achieves asymptotically negligible noise relative to statistical error, demonstrating superior utility in linear and quantile regression while maintaining strong privacy guarantees.

ABSTRACT

This paper presents a new mechanism for producing sanitized statistical summaries that achieve \emph{differential privacy}, called the \emph{K-Norm Gradient} Mechanism, or KNG. This new approach maintains the strong flexibility of the exponential mechanism, while achieving the powerful utility performance of objective perturbation. KNG starts with an inherent objective function (often an empirical risk), and promotes summaries that are close to minimizing the objective by weighting according to how far the gradient of the objective function is from zero. Working with the gradient instead of the original objective function allows for additional flexibility as one can penalize using different norms. We show that, unlike the exponential mechanism, the noise added by KNG is asymptotically negligible compared to the statistical error for many problems. In addition to theoretical guarantees on privacy and utility, we confirm the utility of KNG empirically in the settings of linear and quantile regression through simulations.

Motivation & Objective

• To address the high noise magnitude of the exponential mechanism in private statistical estimation, which often matches the statistical error rate.
• To develop a mechanism that retains the flexibility of the exponential mechanism while achieving utility comparable to objective perturbation.
• To provide a unified framework that generalizes existing mechanisms like Laplace, K-norm, and PrivateQuantile via a flexible norm-based gradient weighting.
• To empirically validate KNG’s performance in linear and quantile regression, showing it approaches the non-private error rate asymptotically.
• To explore the potential of KNG in complex parameter spaces such as Hilbert and Riemannian manifolds, where gradients are well-defined.

Proposed method

• KNG constructs a probability density proportional to exp(−c₀‖∇ℓₙ(θ;D)‖_K), where ∇ℓₙ is the gradient of the empirical objective function and ‖·‖_K is a user-chosen K-norm.
• The K-norm is defined via a convex set K, which can be tailored to the problem’s sensitivity structure, enabling norm-based utility optimization.
• The mechanism promotes solutions θ where the gradient of the objective is close to zero, effectively targeting minimizers of the objective function.
• KNG is shown to be an instance of the exponential mechanism, but with improved statistical efficiency due to gradient-based weighting.
• For sampling, KNG uses Markov Chain Monte Carlo (MCMC) methods, specifically one-at-a-time Gibbs sampling, which is effective in high-dimensional settings.
• The method is applied to mean estimation, quantile estimation, linear regression, and quantile regression, with empirical validation at ε=1.

Experimental results

Research questions

• RQ1 Can a differentially private mechanism be designed that maintains the flexibility of the exponential mechanism while achieving utility comparable to objective perturbation?
• RQ2 Does using the gradient of the objective function with a norm-based weighting reduce noise magnitude relative to statistical error?
• RQ3 Can KNG be efficiently sampled and applied to regression problems, including quantile regression, where no prior private mechanism exists?
• RQ4 Does KNG achieve oₚ(n⁻¹/²) error in non-strongly convex settings, suggesting broader applicability than theoretical assumptions imply?
• RQ5 Can KNG be extended to infinite-dimensional parameter spaces such as Hilbert or Riemannian manifolds?

Key findings

• KNG achieves asymptotically negligible noise relative to statistical error in many problems, unlike the exponential mechanism, which adds noise of the same order as estimation error.
• In simulations of linear and quantile regression, KNG’s estimation error approaches the non-private estimator’s error rate (Oₚ(n⁻¹/²)), indicating efficient private estimation.
• The exponential mechanism’s error approaches a line with slope −1/2 but higher intercept, indicating higher asymptotic variance, while KNG’s error aligns closely with the non-private line.
• KNG is the first mechanism for private quantile regression, empirically shown to be efficient and practical using MCMC sampling.
• Despite theoretical assumptions in Theorem 3.2 not being satisfied in the non-convex regression setting, KNG still achieves oₚ(n⁻¹/²) error, suggesting broader applicability.
• KNG generalizes existing mechanisms: Laplace, K-norm, and PrivateQuantile mechanisms are all special cases of KNG with specific norm choices.