[Paper Review] Differentially Private Empirical Risk Minimization Revisited: Faster and More General

Di Wang, Minwei Ye+1 more|arXiv (Cornell University)|Feb 14, 2018

Complexity and Algorithms in Graphs141 citations

TL;DR

The paper revisits DP-ERM for convex and non-convex losses, presenting faster gradient-perturbation algorithms with improved utility bounds and extending results to Polyak-Lojasiewicz conditions.

ABSTRACT

In this paper we study the differentially private Empirical Risk Minimization (ERM) problem in different settings. For smooth (strongly) convex loss function with or without (non)-smooth regularization, we give algorithms that achieve either optimal or near optimal utility bounds with less gradient complexity compared with previous work. For ERM with smooth convex loss function in high-dimensional ($p\\gg n$) setting, we give an algorithm which achieves the upper bound with less gradient complexity than previous ones. At last, we generalize the expected excess empirical risk from convex loss functions to non-convex ones satisfying the Polyak-Lojasiewicz condition and give a tighter upper bound on the utility than the one in \\cite{ijcai2017-548}.

Motivation & Objective

• Motivate efficient differential privacy for empirical risk minimization (ERM) on sensitive data.
• Improve utility bounds and gradient complexity for DP-ERM across convex, strongly convex, and non-convex settings.
• Extend DP-ERM analysis to non-convex losses satisfying the Polyak-Lojasiewicz condition.
• Address high-dimensional ERM by reducing dependence on dimension via geometry of the constraint set.

Proposed method

• Propose DP-SVRG and DP-SVRG++ variants with Gaussian gradient perturbation to achieve private proximal updates.
• Establish privacy via Gaussian mechanism and moments accountant, with bounds on noise scale csigma^2 proportional to G^2Tm/n^2psilon^2.
• Show improved utility bounds for strongly convex and non-strongly convex cases, with gradient complexity analyses.
• Introduce DP-AccMD to exploit Gaussian width and Minkowski norms for high-dimensional, geometry-aware DP-ERM.
• Extend analysis to non-convex objectives under the Polyak-Lojasiewicz (PL) condition and provide tighter utility bounds.

Experimental results

Research questions

• RQ1 How can gradient-perturbation DP-ERM achieve better utility and lower gradient complexity for convex and strongly convex losses?
• RQ2 Can DP-ERM be efficiently extended to high-dimensional settings with reduced dimension dependence via geometric constraints?
• RQ3 What are the utility guarantees for non-strongly convex and non-convex losses under DP, particularly under Polyak-Lojasiewicz conditions?
• RQ4 Does DP-AccMD offer faster convergence by leveraging Gaussian width and Minkowski norms in private ERM?
• RQ5 How do privacy accounting techniques (Gaussian mechanism, moments accountant, advanced composition) influence practical DP-ERM algorithms?

Key findings

• DP-SVRG achieves near-optimal utility with reduced gradient complexity in the strongly convex case compared to prior DP methods.
• DP-SVRG++ attains near-optimal utility for non-strongly convex cases with gradient complexity improved to O(n^{1.5}) in high dimensions.
• DP-AccMD provides privacy guarantees with Gaussian noise and achieves utility bounds that scale with Gaussian width G_C and diameter ||C||_2 rather than the ambient dimension.
• For non-convex objectives satisfying the Polyak-Lojasiewicz condition, DP-GD yields near-optimal excess empirical risk with favorable dependence on n and p.
• The framework unifies DP-ERM across smooth/convex, high-dimensional, and certain non-convex settings, with tighter bounds than previous results.