[Paper Review] A Parallel SGD method with Strong Convergence

Dhruv Mahajan, S. Sathiya Keerthi+2 more|arXiv (Cornell University)|Nov 4, 2013

Face and Expression Recognition12 references13 citations

TL;DR

This paper proposes a novel parallel stochastic gradient descent (SGD) method that achieves strong global linear convergence by using locally optimized approximations of the objective function on each node, with gradient consistency enforced via a tilt term. The method outperforms existing distributed methods in early convergence and generalization performance, especially on large-scale, high-dimensional datasets like kdd2010.

ABSTRACT

This paper proposes a novel parallel stochastic gradient descent (SGD) method that is obtained by applying parallel sets of SGD iterations (each set operating on one node using the data residing in it) for finding the direction in each iteration of a batch descent method. The method has strong convergence properties. Experiments on datasets with high dimensional feature spaces show the value of this method.

Motivation & Objective

• To address the limitations of existing parallel SGD methods in distributed learning, particularly poor convergence and high variance due to inconsistent local approximations.
• To develop a distributed optimization method that maintains strong convergence guarantees while enabling efficient parallel computation across multiple nodes.
• To improve early convergence and generalization performance in large-scale linear classification tasks with high-dimensional feature spaces.
• To reduce the number of communication passes required to reach optimal performance in distributed training settings.

Proposed method

• The method uses a batch-level descent framework where each iteration computes a global gradient $ g^r $ at the current weight $ w^r $, which is communicated to all $ P $ nodes.
• Each node $ p $ constructs a locally accurate approximation $ \hat{f}_p(w) $ of the global objective $ f(w) $, ensuring gradient consistency at $ w^r $ via the term $ (g^r - \lambda w^r - \nabla L_p(w^r)) \cdot (w - w^r) $.
• Each node performs $ s $ epochs of SGD on its local approximation $ \hat{f}_p $, starting from $ w^r $, to compute a local descent direction $ d_p $.
• The global search direction $ d^r $ is formed as a convex combination of the individual $ d_p $, ensuring descent if all $ d_p $ are descent directions.
• A line search satisfying Armijo-Wolfe conditions is used to determine the step size $ t $, ensuring sufficient decrease and curvature in the global objective.
• The method integrates a trust-region-like line search and uses a communication-efficient AllReduce architecture for aggregation across nodes.

Experimental results

Research questions

• RQ1 Can a parallel SGD method be designed to achieve strong global linear convergence while maintaining computational efficiency in distributed settings?
• RQ2 How can local approximations of the global objective function be constructed to ensure gradient consistency and descent direction validity across nodes?
• RQ3 What is the impact of the number of local SGD epochs $ s $ on the convergence rate and communication efficiency of the method?
• RQ4 How does the proposed method compare to state-of-the-art distributed methods like SQM and Hybrid in terms of convergence speed and generalization performance?
• RQ5 Under what conditions does the method maintain strong convergence when the global objective is non-convex, such as in deep learning?

Key findings

• The proposed method achieves global linear convergence (glrc), ensuring a predictable and fast decrease in the objective function gap $ (f - f^*) $.
• On the kdd2010 dataset with 8.41 million examples, the method required significantly fewer communication passes than SQM and Hybrid to reach the same objective function accuracy.
• The method reached stable generalization performance (measured by AUPRC) much faster than SQM and Hybrid, especially in early training stages.
• When the number of nodes increased from 25 to 100, the performance gap between the proposed method and SQM/Hybrid narrowed, due to reduced approximation quality of $ \hat{f}_p $ in high-node regimes.
• The method’s performance was sensitive to the number of local SGD epochs $ s $, with optimal $ s $ balancing local optimization and global convergence speed.
• The method outperformed SQM and Hybrid in both objective function reduction and AUPRC, particularly in scenarios with high-dimensional data and limited communication budget.