[Paper Review] D$^2$: Decentralized Training over Decentralized Data

Hanlin Tang, Xiangru Lian+3 more|arXiv (Cornell University)|Mar 19, 2018

Stochastic Gradient Optimization Techniques35 references185 citations

TL;DR

D$^2$ is a variance-reduction extension of D-PSGD that is robust to high data variance across workers, achieving faster convergence than D-PSGD and approaching centralized SGD performance.

ABSTRACT

While training a machine learning model using multiple workers, each of which collects data from their own data sources, it would be most useful when the data collected from different workers can be {\em unique} and {\em different}. Ironically, recent analysis of decentralized parallel stochastic gradient descent (D-PSGD) relies on the assumption that the data hosted on different workers are {\em not too different}. In this paper, we ask the question: {\em Can we design a decentralized parallel stochastic gradient descent algorithm that is less sensitive to the data variance across workers?} In this paper, we present D$^2$, a novel decentralized parallel stochastic gradient descent algorithm designed for large data variance \xr{among workers} (imprecisely, "decentralized" data). The core of D$^2$ is a variance blackuction extension of the standard D-PSGD algorithm, which improves the convergence rate from $O\left({σ\over \sqrt{nT}} + {(nζ^2)^{\frac{1}{3}} \over T^{2/3}} ight)$ to $O\left({σ\over \sqrt{nT}} ight)$ where $ζ^{2}$ denotes the variance among data on different workers. As a result, D$^2$ is robust to data variance among workers. We empirically evaluated D$^2$ on image classification tasks where each worker has access to only the data of a limited set of labels, and find that D$^2$ significantly outperforms D-PSGD.

Motivation & Objective

• Motivate decentralized training when data across workers is highly non-identical.
• Develop a variance-reduction mechanism integrated with D-PSGD to reduce outer variance impact.
• Theoretically establish convergence guarantees showing improved rates.
• Empirically validate D$^2$ on image classification tasks with non-uniform label distributions.

Proposed method

• Extend D-PSGD with a variance-reduction component by storing gradients and local models from the last iterate and linearly combining them with current gradients and models.
• Update rule aggregates local updates with a combination of current and previous gradients to mitigate data variance across workers.
• Provide a global update expression X_{t+1} = (2X_t - X_{t-1} - γG(X_t; ξ_t) + γG(X_{t-1}; ξ_{t-1}))W.
• Show that the mean iterate follows a reduced-variance dynamic, leading to improved convergence independent of the global data variance ζ^2 across workers.
• State assumptions including Lipschitz gradients, bounded per-worker variance, symmetric consensus matrix with spectral gap, and network topology considerations.
• Deliver theoretical convergence guarantees and corollaries demonstrating rate improvements over D-PSGD.

Experimental results

Research questions

• RQ1 Can a decentralized SGD algorithm be designed to be robust to large data variance among workers?
• RQ2 Does a variance-reduction strategy integrated into D-PSGD improve convergence rates from O(σ/√(nT)) + O((nζ^2)^{1/3}/T^{2/3}) to O(σ/√(nT))?
• RQ3 Under what conditions does D$^2$ achieve linear speedup with respect to the number of workers?
• RQ4 How does D$^2$ perform empirically when workers hold non-overlapping or limited-label datasets compared to D-PSGD and centralized SGD?

Key findings

• D$^2$ achieves a convergence rate of O(σ/√(nT)) compared to D-PSGD's rate that depends on ζ^2 (outer variance).
• The variance-reduction component eliminates dependence on the global data variance across workers in the asymptotic rate.
• Theoretical results show linear speedup with the number of workers under suitable conditions.
• Empirical experiments on image classification tasks with restricted-label data per worker show D$^2$ significantly outperforms D-PSGD, approaching centralized performance, especially in unshuffled (high-variance) setups.