[Paper Review] Differentially Private Coordinate Descent for Composite Empirical Risk Minimization

Paul Mangold, Aurélien Bellet+2 more|arXiv (Cornell University)|Oct 22, 2021

Privacy-Preserving Technologies in Data44 references4 citations

TL;DR

This paper introduces Differentially Private Coordinate Descent (DP-CD), a novel algorithm for composite Empirical Risk Minimization under differential privacy. By leveraging coordinate-wise gradient perturbations and large constant step sizes, DP-CD achieves better utility than DP-SGD, especially in imbalanced problems, and matches theoretical lower bounds up to logarithmic factors.

ABSTRACT

Machine learning models can leak information about the data used to train them. To mitigate this issue, Differentially Private (DP) variants of optimization algorithms like Stochastic Gradient Descent (DP-SGD) have been designed to trade-off utility for privacy in Empirical Risk Minimization (ERM) problems. In this paper, we propose Differentially Private proximal Coordinate Descent (DP-CD), a new method to solve composite DP-ERM problems. We derive utility guarantees through a novel theoretical analysis of inexact coordinate descent. Our results show that, thanks to larger step sizes, DP-CD can exploit imbalance in gradient coordinates to outperform DP-SGD. We also prove new lower bounds for composite DP-ERM under coordinate-wise regularity assumptions, that are nearly matched by DP-CD. For practical implementations, we propose to clip gradients using coordinate-wise thresholds that emerge from our theory, avoiding costly hyperparameter tuning. Experiments on real and synthetic data support our results, and show that DP-CD compares favorably with DP-SGD.

Motivation & Objective

• Address the limitations of DP-SGD in terms of step size constraints and sensitivity to gradient imbalance in differentially private optimization.
• Develop a new differentially private optimization framework tailored for composite ERM problems with nonsmooth regularizers like ℓ1.
• Establish tight theoretical bounds on privacy-utility trade-offs by analyzing inexact proximal coordinate descent with perturbed gradients.
• Provide practical guidelines for hyperparameter tuning by deriving coordinate-wise clipping thresholds from theory, avoiding costly manual calibration.
• Demonstrate the superiority of DP-CD over DP-SGD in both balanced and imbalanced settings through extensive experiments on real and synthetic data.

Proposed method

• Propose a differentially private proximal coordinate descent algorithm (DP-CD) that updates one coordinate at a time using perturbed partial gradients.
• Use coordinate-wise gradient clipping based on theoretical smoothness constants to reduce noise while preserving privacy.
• Introduce a novel recursion analysis of iterates' distance to the optimal solution, tracking coordinate-wise regularity constants for tighter convergence bounds.
• Derive privacy guarantees via the advanced composition theorem, leveraging the lower sensitivity of partial gradients compared to full gradients.
• Estimate coordinate-wise smoothness constants privately using a subsampled, differentially private procedure to enable deployment without prior knowledge.
• Implement a single-hyperparameter rule to adapt coordinate-wise clipping thresholds, reducing tuning complexity.

Experimental results

Research questions

• RQ1 Can coordinate descent methods achieve better utility than SGD-style methods in differentially private ERM under the same privacy budget?
• RQ2 How does the imbalance in coordinate-wise gradient magnitudes affect the performance of private optimization algorithms?
• RQ3 What are the optimal privacy-utility trade-offs for composite DP-ERM under coordinate-wise regularity assumptions?
• RQ4 Can a private coordinate descent algorithm match known theoretical lower bounds for DP-ERM up to logarithmic factors?
• RQ5 How can coordinate-wise clipping thresholds be set efficiently without extensive hyperparameter tuning in practice?

Key findings

• DP-CD achieves significantly lower optimization error than DP-SGD on imbalanced datasets, with relative error reductions of up to 90% on the California housing dataset (e.g., 0.0010 vs. 0.0087 at ε=1).
• On the sparse LASSO problem, DP-CD reduces error by over 60% compared to DP-SGD at ε=10, achieving 0.2498 vs. 0.7551 in relative error.
• DP-CD runs approximately 4–5 times faster than DP-SGD per iteration, enabling faster convergence to high-utility solutions.
• The algorithm matches theoretical lower bounds for composite DP-ERM under coordinate-wise regularity up to logarithmic factors, indicating near-optimality.
• The proposed single-hyperparameter rule for coordinate-wise clipping enables effective performance without manual tuning, validated across multiple datasets.
• Even in balanced settings, DP-CD outperforms DP-SGD, confirming its broader relevance beyond imbalanced problems.