[Paper Review] Stochastic Adaptive Line Search for Differentially Private Optimization

Chen Chen, Jae Wook Lee|arXiv (Cornell University)|Aug 18, 2020

Privacy-Preserving Technologies in Data26 references4 citations

TL;DR

This paper proposes a Rényi differentially private stochastic gradient descent algorithm with adaptive step size selection via a stochastic Armijo line search, using noisy gradients and function estimates to satisfy the Armijo condition while minimizing privacy budget usage. It introduces adaptive privacy budget allocation based on gradient direction changes and achieves superior convergence and utility compared to existing private optimizers on both convex and non-convex problems.

ABSTRACT

The performance of private gradient-based optimization algorithms is highly dependent on the choice of step size (or learning rate) which often requires non-trivial amount of tuning. In this paper, we introduce a stochastic variant of classic backtracking line search algorithm that satisfies Rényi differential privacy. Specifically, the proposed algorithm adaptively chooses the step size satsisfying the the Armijo condition (with high probability) using noisy gradients and function estimates. Furthermore, to improve the probability with which the chosen step size satisfies the condition, it adjusts per-iteration privacy budget during runtime according to the reliability of noisy gradient. A naive implementation of the backtracking search algorithm may end up using unacceptably large privacy budget as the ability of adaptive step size selection comes at the cost of extra function evaluations. The proposed algorithm avoids this problem by using the sparse vector technique combined with the recent privacy amplification lemma. We also introduce a privacy budget adaptation strategy in which the algorithm adaptively increases the budget when it detects that directions pointed by consecutive gradients are drastically different. Extensive experiments on both convex and non-convex problems show that the adaptively chosen step sizes allow the proposed algorithm to efficiently use the privacy budget and show competitive performance against existing private optimizers.

Motivation & Objective

• To address the challenge of step size selection in differentially private optimization, which critically affects convergence and utility.
• To design a privacy-preserving line search mechanism that adaptively selects step sizes satisfying the Armijo condition under Rényi differential privacy.
• To reduce excessive privacy budget consumption from multiple function evaluations in backtracking line search by leveraging the sparse vector technique and privacy amplification.
• To improve reliability of step size selection under noisy gradients by dynamically adjusting the privacy budget when consecutive gradients point in drastically different directions.
• To empirically validate that adaptive step size selection enhances performance across convex, non-convex, and deep learning problems under differential privacy.

Proposed method

• Proposes a stochastic backtracking line search algorithm that uses noisy gradients and function estimates to satisfy the Armijo condition with high probability under Rényi differential privacy.
• Applies the sparse vector technique to limit privacy budget usage by only paying for iterations where the Armijo condition is checked, rather than for all evaluations.
• Employs privacy amplification via random subsampling to relax the privacy budget constraint on function evaluations.
• Introduces a dynamic privacy budget adaptation strategy that increases the budget when the angle between consecutive gradients exceeds a threshold, indicating unstable or conflicting gradient directions.
• Uses a Gaussian mechanism with calibrated noise scale based on the per-iteration privacy budget to ensure Rényi differential privacy.
• Implements a NoisyBTLS subroutine that returns the largest valid step size within a predefined range, with privacy budget allocated to gradient and function evaluations.

Experimental results

Research questions

• RQ1 Can a differentially private line search algorithm be designed that adaptively selects step sizes satisfying the Armijo condition while minimizing privacy budget usage?
• RQ2 How can the reliability of step size selection be improved in the presence of noisy gradients and function estimates under differential privacy?
• RQ3 What is the impact of adaptive privacy budget allocation on convergence speed and model utility in private optimization?
• RQ4 How does the proposed method compare to existing private optimizers like DP-SGD and DP-Adam in terms of convergence and final model performance?
• RQ5 Can adaptive step size selection via line search improve performance on both convex and non-convex machine learning problems under differential privacy?

Key findings

• The proposed DP-BLSGD algorithm outperforms DP-SGD and DP-Adam in convergence speed and final accuracy on MLP and CNN models for MNIST, FMNIST, and Cifar-10, especially in objective value reduction.
• On logistic regression tasks, DP-BLSGD achieves higher classification accuracy than DP-AGD and Outpert-RSGD on 3 out of 4 datasets, particularly under low privacy budgets.
• The algorithm's adaptive step size selection leads to faster convergence, with step sizes decreasing appropriately over training iterations, as observed in the bottom row of Figure 5.
• When privacy budget is not increased during training, DP-BLSGD still achieves better performance than DP-SGD and DP-Adam, demonstrating the value of line search alone.
• The dynamic privacy budget adaptation strategy improves reliability: when gradient directions diverge significantly, the algorithm increases budget to ensure stable gradient estimation.
• On the Bank dataset, Outpert-RSGD outperforms DP-BLSGD, likely due to the dataset's small training range, which may reduce the effectiveness of gradient perturbation-based methods.