[Paper Review] Distributed Stochastic Optimization via Adaptive Stochastic Gradient Descent.

Ashok Cutkosky, Róbert Busa‐Fekete|arXiv (Cornell University)|Feb 16, 2018

Stochastic Gradient Optimization Techniques17 references2 citations

TL;DR

This paper proposes a distributed stochastic optimization method based on adaptive step sizes and variance reduction that achieves linear speedup across machines, minimal synchronization rounds (logarithmic in dataset size), and low memory usage. It generalizes any serial SGD algorithm, enabling efficient parallelization of adaptive SGD methods with significant performance gains on large-scale logistic regression in Spark.

ABSTRACT

Stochastic convex optimization algorithms are the most popular way to train machine learning models on large-scale data. Scaling up the training process of these models is crucial in many applications, but the most popular algorithm, Stochastic Gradient Descent (SGD), is a serial algorithm that is surprisingly hard to parallelize. In this paper, we propose an efficient distributed stochastic optimization method based on adaptive step sizes and variance reduction techniques. We achieve a linear speedup in the number of machines, small memory footprint, and only a small number of synchronization rounds -- logarithmic in dataset size -- in which the computation nodes communicate with each other. Critically, our approach is a general reduction than parallelizes any serial SGD algorithm, allowing us to leverage the significant progress that has been made in designing adaptive SGD algorithms. We conclude by implementing our algorithm in the Spark distributed framework and exhibit dramatic performance gains on large-scale logistic regression problems.

Motivation & Objective

• To address the challenge of efficiently parallelizing serial Stochastic Gradient Descent (SGD) for large-scale machine learning.
• To reduce synchronization overhead in distributed optimization by limiting communication rounds to logarithmic scale in dataset size.
• To maintain low memory usage while scaling across multiple machines.
• To generalize the approach so it can parallelize any existing serial adaptive SGD algorithm.
• To demonstrate practical performance gains on real-world large-scale logistic regression problems.

Proposed method

• The method employs adaptive step sizes to improve convergence per iteration, leveraging advances in adaptive SGD algorithms.
• It integrates variance reduction techniques to stabilize training and accelerate convergence in distributed settings.
• The algorithm achieves linear speedup by minimizing the number of synchronization rounds, which scales logarithmically with dataset size.
• Communication between machines is optimized through a general reduction mechanism that parallelizes any serial SGD implementation.
• The approach maintains a small memory footprint by avoiding storage of full gradients or large historical buffers.
• The method is implemented in the Apache Spark framework to enable practical deployment on large clusters.

Experimental results

Research questions

• RQ1 Can adaptive stochastic gradient descent be efficiently parallelized in a distributed setting with minimal synchronization?
• RQ2 Does the proposed method achieve linear speedup with respect to the number of machines in distributed training?
• RQ3 Can variance reduction and adaptive step sizes be combined effectively in a distributed framework to improve convergence?
• RQ4 How does the communication overhead scale with dataset size in the proposed distributed optimization framework?
• RQ5 To what extent can the method generalize to any serial SGD algorithm without sacrificing performance?

Key findings

• The proposed method achieves linear speedup in the number of machines, significantly reducing training time on large-scale datasets.
• Synchronization rounds scale logarithmically with dataset size, minimizing communication bottlenecks in distributed training.
• The method maintains a small memory footprint, making it suitable for resource-constrained distributed environments.
• The algorithm successfully generalizes any serial adaptive SGD, enabling the use of advanced adaptive methods in distributed settings.
• In Spark-based experiments, the method demonstrates dramatic performance gains on large-scale logistic regression problems compared to standard distributed SGD.