[论文解读] Defending Against Saddle Point Attack in Byzantine-Robust Distributed Learning

Dong Yin, Yudong Chen+2 more|arXiv (Cornell University)|Jun 14, 2018

Stochastic Gradient Optimization Techniques被引用 47

一句话总结

介绍 ByzantinePGD，一种健壮的第一阶算法，能够在非凸分布式学习中逃离由拜占庭工作者造成的鞍点和伪局部极小值，并具备理论保证和实用的鲁棒梯度估计器。

ABSTRACT

We study robust distributed learning that involves minimizing a non-convex loss function with saddle points. We consider the Byzantine setting where some worker machines have abnormal or even arbitrary and adversarial behavior. In this setting, the Byzantine machines may create fake local minima near a saddle point that is far away from any true local minimum, even when robust gradient estimators are used. We develop ByzantinePGD, a robust first-order algorithm that can provably escape saddle points and fake local minima, and converge to an approximate true local minimizer with low iteration complexity. As a by-product, we give a simpler algorithm and analysis for escaping saddle points in the usual non-Byzantine setting. We further discuss three robust gradient estimators that can be used in ByzantinePGD, including median, trimmed mean, and iterative filtering. We characterize their performance in concrete statistical settings, and argue for their near-optimality in low and high dimensional regimes.

研究动机与目标

• Motivate robust distributed optimization for non-convex losses under Byzantine faults.
• Develop an algorithm that escapes saddle points despite adversarial gradients.
• Provide theoretical guarantees for convergence to approximate local minima under inexact gradient oracles.
• Propose and analyze robust gradient aggregation methods suitable for Byzantine environments.

提出的方法

• Propose ByzantinePGD which aggregates gradients via a GradAGG oracle to obtain a Delta-inexact gradient.
• Incorporate random perturbations to iterates to escape saddle points and fake local minima.
• Use multiple rounds of calibrated perturbation and distance-based escape criteria instead of relying on function values.
• Provide a two-part framework separating optimization (inexact gradient descent) and statistics (robust gradient estimation).
• Characterize iteration complexity as roughly matching non-Byzantine GD for non-convex problems, up to log factors.
• Show how three robust gradient estimators (median, trimmed mean, iterative filtering) yield concrete statistical guarantees.

实验结果

研究问题

• RQ1 Can we achieve provable escape from saddle points in the presence of Byzantine workers while minimizing communication and computation?
• RQ2 How do robust gradient aggregation methods affect the inexact gradient oracle and overall convergence in non-convex distributed learning?
• RQ3 What are the theoretical limits for first- and second-order stationarity under Byzantine adversaries in distributed non-convex optimization?
• RQ4 How do median, trimmed mean, and iterative filtering perform in high- and low-dimensional regimes under Byzantine faults?
• RQ5 Is it possible to obtain convergence guarantees without requiring function value evaluations?

主要发现

• ByzantinePGD achieves escape from saddle points and converges to an approximate local minimizer under a Delta-inexact gradient oracle.
• The algorithm uses multiple perturbation rounds and a distance-based escape criterion, yielding a simpler analysis than prior PGD variants.
• Three robust aggregation schemes (median, trimmed mean, iterative filtering) provide concrete statistical guarantees for the gradient error Delta.
• With Delta-inexact gradients, the method attains an O(1/Delta^2) iteration complexity to reach a first-order stationarity and a softened second-order condition.
• Lower bounds indicate no algorithm can achieve substantially better second-order guarantees than O(Delta^1/2) in this setting.
• The results extend beyond Byzantine distributed learning to any non-convex optimization with inexact gradients, including noisy-but-non-adversarial settings.