[Paper Review] Distributed Stochastic Variance Reduced Gradient Methods.

Jason D. Lee, Tengyu Ma+1 more|arXiv (Cornell University)|Jul 27, 2015

Stochastic Gradient Optimization Techniques10 references10 citations

TL;DR

This paper proposes a distributed stochastic variance reduced gradient (DSVRG) method for minimizing the average of convex functions in large-scale machine learning. It achieves optimal communication efficiency and runtime by minimizing rounds of communication, matching a proven lower bound, and outperforming existing methods when the condition number is not excessively large relative to local data size.

ABSTRACT

We study distributed optimization algorithms for minimizing the average of convex functions. The applications include empirical risk minimization problems in statistical machine learning where the datasets are large and have to be stored on different machines. We design a distributed stochastic variance reduced gradient algorithm that, under certain conditions on the condition number, simultaneously achieves the optimal parallel runtime, amount of communication and rounds of communication among all distributed first-order methods up to constant factors. Our method and its accelerated extension also outperform existing distributed algorithms in terms of the rounds of communication as long as the condition number is not too large compared to the size of data in each machine. We also prove a lower bound for the number of rounds of communication for a broad class of distributed first-order methods including the proposed algorithms in this paper. We show that our accelerated distributed stochastic variance reduced gradient algorithm achieves this lower bound so that it uses the fewest rounds of communication among all distributed first-order algorithms.

Motivation & Objective

• To design a distributed first-order optimization method that minimizes the average of convex functions with optimal communication efficiency.
• To achieve optimal parallel runtime, communication volume, and number of communication rounds among distributed first-order methods.
• To analyze the fundamental limits of communication efficiency by proving a lower bound for a broad class of distributed first-order methods.
• To develop an accelerated variant that matches the lower bound and outperforms existing algorithms in communication rounds.

Proposed method

• Proposes a distributed stochastic variance reduced gradient (DSVRG) algorithm tailored for empirical risk minimization on distributed datasets.
• Uses variance reduction techniques to stabilize gradient updates and reduce noise in distributed settings.
• Designs the algorithm to minimize the number of communication rounds while maintaining optimal convergence rates.
• Introduces an accelerated extension of DSVRG that achieves the theoretical lower bound on communication rounds.
• Analyzes the condition number's role in determining communication efficiency and convergence speed.
• Employs a theoretical framework to derive a lower bound on communication rounds for a broad class of distributed first-order methods.

Experimental results

Research questions

• RQ1 Can a distributed first-order method achieve optimal communication efficiency in terms of rounds, volume, and runtime?
• RQ2 What is the fundamental lower bound on the number of communication rounds for distributed first-order optimization?
• RQ3 How does the condition number affect the communication efficiency of distributed optimization algorithms?
• RQ4 Can an accelerated variant of DSVRG match the theoretical lower bound on communication rounds?
• RQ5 How does the proposed method compare to existing distributed algorithms in terms of communication complexity?

Key findings

• The proposed DSVRG algorithm achieves optimal parallel runtime, communication volume, and number of communication rounds up to constant factors.
• The accelerated DSVRG variant matches the derived lower bound on communication rounds, making it communication-optimal.
• The method outperforms existing distributed algorithms in terms of communication rounds when the condition number is not too large relative to local data size.
• The paper establishes a theoretical lower bound for communication rounds that applies to a broad class of distributed first-order methods.
• The algorithm maintains optimal convergence rates under standard assumptions on convexity and smoothness.
• The results demonstrate that communication efficiency can be maximized by balancing condition number and local data size.