[Paper Review] Stochastic Sign Descent Methods: New Algorithms and Better Theory

Mher Safaryan, Peter Richtárik|arXiv (Cornell University)|May 30, 2019

Stochastic Gradient Optimization Techniques36 references4 citations

TL;DR

This paper introduces Stochastic Sign Descent with Momentum (SSDM), a novel sign-based optimization method that achieves convergence in distributed training under standard bounded variance assumptions, resolving a fundamental flaw in signSGD. It proposes a new gradient norm and theoretical framework enabling optimal convergence rates with 1-bit compression and small mini-batches.

ABSTRACT

Various gradient compression schemes have been proposed to mitigate the communication cost in distributed training of large scale machine learning models. Sign-based methods, such as signSGD, have recently been gaining popularity because of their simple compression rule and connection to adaptive gradient methods, like ADAM. In this paper, we analyze sign-based methods for non-convex optimization in three key settings: (i) standard single node, (ii) parallel with shared data and (iii) distributed with partitioned data. For single machine case, we generalize the previous analysis of signSGD relying on intuitive bounds on success probabilities and allowing even biased estimators. Furthermore, we extend the analysis to parallel setting within a parameter server framework, where exponentially fast noise reduction is guaranteed with respect to number of nodes, maintaining $1$-bit compression in both directions and using small mini-batch sizes. Next, we identify a fundamental issue with signSGD to converge in distributed environment. To resolve this issue, we propose a new sign-based method, {\em Stochastic Sign Descent with Momentum (SSDM)}, which converges under standard bounded variance assumption with the optimal asymptotic rate. We validate several aspects of our theoretical findings with numerical experiments.

Motivation & Objective

• Address the lack of theoretical convergence guarantees for signSGD in distributed settings with partitioned data.
• Develop a sign-based method that maintains 1-bit compression while achieving optimal convergence rates under standard assumptions.
• Provide a unified theoretical framework for sign-based methods across single-node, parallel, and distributed settings.
• Overcome the issue of biased gradient estimators and non-uniform success probabilities in sign-based optimization.
• Validate theoretical claims with experiments on Rosenbrock function and MNIST using FNNs.

Proposed method

• Proposes a new ρ-norm for measuring gradients, which generalizes the l¹ and l² norms and enables tighter convergence analysis.
• Introduces Stochastic Sign Descent with Momentum (SSDM), a novel algorithm that combines sign-based updates with momentum to stabilize training.
• Uses success probability bounds (SPB) as a key theoretical tool to analyze convergence under non-i.i.d. and skewed gradient distributions.
• Applies a parameter server framework to model parallel and distributed training, enabling exponential noise reduction with increasing nodes.
• Derives convergence rates under bounded variance and mild smoothness assumptions, with explicit dependence on gradient success probabilities.
• Employs a majority vote mechanism in distributed settings to improve robustness, while maintaining 1-bit communication in both directions.

Experimental results

Research questions

• RQ1 Can sign-based methods achieve optimal convergence rates in distributed training with partitioned data and 1-bit compression?
• RQ2 What are the theoretical conditions under which signSGD fails to converge, and how can this be remedied?
• RQ3 How do success probability bounds (SPB) influence the convergence behavior of sign-based methods?
• RQ4 Can a momentum-based sign method maintain convergence under standard bounded variance assumptions while handling biased estimators?
• RQ5 How does the choice of mini-batch size and gradient distribution shape the convergence dynamics of sign-based methods?

Key findings

• SSDM achieves a convergence rate of O(1/K¹/⁴) under standard bounded variance assumptions, matching the best-known rates for non-convex optimization.
• The proposed ρ-norm enables convergence analysis for sign-based methods even with biased gradient estimators and non-i.i.d. gradients.
• In the parallel setting with shared data, exponential noise reduction is achieved with respect to the number of nodes, maintaining 1-bit compression.
• Numerical experiments confirm that signSGD can converge under multi-modal and skewed gradient distributions when success probability bounds are satisfied.
• The method shows improved early training efficiency compared to SGD, particularly when success probabilities are high and mini-batch sizes are small.
• Larger mini-batch sizes lead to better convergence, consistent with theoretical bounds on success probability and mini-batch size dependencies.