[Paper Review] Differentially Private Empirical Risk Minimization with Sparsity-Inducing Norms

K.S. Kumar, Marc Peter Deisenroth|arXiv (Cornell University)|May 13, 2019

Privacy-Preserving Technologies in Data21 references4 citations

TL;DR

This paper proposes differentially private empirical risk minimization (ERM) algorithms using sparsity-inducing norms, focusing on non-smooth regularizers that induce structured sparsity. It introduces a private Frank-Wolfe algorithm for the dual ERM problem and derives excess risk bounds dependent on the Gaussian width of the dual norm's unit ball, establishing equivalence between objective perturbation and output perturbation in the dual space.

ABSTRACT

Differential privacy is concerned about the prediction quality while measuring the privacy impact on individuals whose information is contained in the data. We consider differentially private risk minimization problems with regularizers that induce structured sparsity. These regularizers are known to be convex but they are often non-differentiable. We analyze the standard differentially private algorithms, such as output perturbation, Frank-Wolfe and objective perturbation. Output perturbation is a differentially private algorithm that is known to perform well for minimizing risks that are strongly convex. Previous works have derived excess risk bounds that are independent of the dimensionality. In this paper, we assume a particular class of convex but non-smooth regularizers that induce structured sparsity and loss functions for generalized linear models. We also consider differentially private Frank-Wolfe algorithms to optimize the dual of the risk minimization problem. We derive excess risk bounds for both these algorithms. Both the bounds depend on the Gaussian width of the unit ball of the dual norm. We also show that objective perturbation of the risk minimization problems is equivalent to the output perturbation of a dual optimization problem. This is the first work that analyzes the dual optimization problems of risk minimization problems in the context of differential privacy.

Motivation & Objective

• Address the challenge of achieving differential privacy in ERM with non-smooth, structured sparsity-inducing regularizers.
• Analyze the utility of output perturbation and Frank-Wolfe algorithms in the context of non-smooth regularizers.
• Provide theoretical excess risk bounds that depend on the Gaussian width of the dual norm's unit ball.
• Establish a novel equivalence between objective perturbation in the primal and output perturbation in the dual optimization problem.
• Open the path for analyzing mirror-descent and gradient-perturbation-based private algorithms in non-smooth ERM settings.

Proposed method

• Apply output perturbation to the primal ERM problem with non-smooth regularizers, adding noise to the model parameters to ensure differential privacy.
• Develop a private Frank-Wolfe algorithm to optimize the dual of the ERM problem, leveraging the structure of the dual space.
• Derive excess risk bounds for both output perturbation and private Frank-Wolfe by analyzing the Gaussian width of the unit ball of the dual norm.
• Use Fenchel duality to reformulate the objective perturbation in the primal as output perturbation in the dual, proving equivalence.
• Utilize strong duality and Fenchel conjugates to transform the perturbed primal problem into a dual optimization problem.
• Leverage submodular functions as regularizers to induce structured sparsity, enabling the use of polyhedral sets in the dual space.

Experimental results

Research questions

• RQ1 How can output perturbation be effectively applied to ERM problems with non-smooth, sparsity-inducing regularizers?
• RQ2 What is the excess risk bound for a private Frank-Wolfe algorithm applied to the dual of an ERM problem with structured sparsity?
• RQ3 Can objective perturbation in the primal ERM problem be equivalently interpreted as output perturbation in the dual problem?
• RQ4 How does the Gaussian width of the dual norm's unit ball influence the excess risk in differentially private ERM?
• RQ5 What structural properties of the dual norm can be exploited to improve utility bounds in private ERM with non-smooth regularizers?

Key findings

• The excess risk bound for output perturbation depends on the Gaussian width of the unit ball of the dual norm, providing a dimension-independent utility guarantee.
• The private Frank-Wolfe algorithm for the dual ERM achieves excess risk bounds that are also governed by the Gaussian width of the dual norm's unit ball.
• Objective perturbation in the primal ERM problem is mathematically equivalent to output perturbation in the dual optimization problem.
• The authors establish the first theoretical analysis of dual optimization problems in the context of differential privacy for ERM.
• The results extend prior work on excess risk bounds by showing that such bounds can be independent of input dimensionality when structured sparsity is induced.
• The framework opens new avenues for analyzing mirror-descent and gradient-perturbation-based private algorithms in non-smooth ERM settings.