[Paper Review] An efficient distributed learning algorithm based on effective local functional approximations

Dhruv Mahajan, Nikunj Agrawal+3 more|arXiv (Cornell University)|Oct 31, 2013

Stochastic Gradient Optimization Techniques27 references6 citations

TL;DR

This paper proposes FADL, a novel distributed learning algorithm that uses local functional approximations to reduce communication overhead in large-scale machine learning. By allowing each node to solve a locally approximated objective and combining results iteratively, FADL achieves global linear convergence and outperforms state-of-the-art methods like Terascale in communication-heavy settings, showing 1–10x speedup on various datasets.

ABSTRACT

Scalable machine learning over big data is an important problem that is receiving a lot of attention in recent years. On popular distributed environments such as Hadoop running on a cluster of commodity machines, communication costs are substantial and algorithms need to be designed suitably considering those costs. In this paper we give a novel approach to the distributed training of linear classifiers (involving smooth losses and L2 regularization) that is designed to reduce the total communication costs. At each iteration, the nodes minimize locally formed approximate objective functions; then the resulting minimizers are combined to form a descent direction to move. Our approach gives a lot of freedom in the formation of the approximate objective function as well as in the choice of methods to solve them. The method is shown to have $O(log(1/ε))$ time convergence. The method can be viewed as an iterative parameter mixing method. A special instantiation yields a parallel stochastic gradient descent method with strong convergence. When communication times between nodes are large, our method is much faster than the Terascale method (Agarwal et al., 2011), which is a state of the art distributed solver based on the statistical query model (Chuet al., 2006) that computes function and gradient values in a distributed fashion. We also evaluate against other recent distributed methods and demonstrate superior performance of our method.

Motivation & Objective

• To address the high communication cost in distributed machine learning over big data, especially when communication latency dominates computation time.
• To design a flexible distributed optimization framework that maintains strong convergence guarantees while minimizing communication passes.
• To enable efficient training of linear classifiers with smooth losses and L2 regularization in decentralized, example-partitioned environments.
• To establish a theoretical foundation for global linear rate of convergence (glrc) in a class of iterative parameter mixing methods.
• To demonstrate superior performance over existing methods like Terascale, CoCoA, and ADMM in terms of total training time and communication efficiency.

Proposed method

• The algorithm uses local functional approximations of the global objective function at each node, based on locally stored data partitions.
• Each node minimizes its own approximated objective function using any suitable local optimization method, allowing flexibility in computation.
• The resulting local minimizers are aggregated to form a descent direction for the global iterate, ensuring descent in the global objective.
• The method is formulated as an iterative parameter mixing (IPM) scheme, with convergence guaranteed under weak convexity and gradient consistency conditions.
• A key innovation is the use of gradient sub-consistency for feature partitioning, enabling overlap in feature sets across nodes without sacrificing convergence.
• The algorithm supports joint example-feature partitioning and can be extended to non-convex settings, though convergence analysis is simpler in convex cases.

Experimental results

Research questions

• RQ1 Can we design a distributed learning algorithm that reduces communication passes while maintaining global linear convergence?
• RQ2 Does allowing local optimization with flexible approximations and early stopping improve total training time in communication-heavy environments?
• RQ3 Can the method be generalized to feature partitioning with overlapping feature sets, preserving convergence?
• RQ4 How does the method compare to state-of-the-art solvers like Terascale, CoCoA, and ADMM in terms of communication and computation trade-offs?
• RQ5 Is there a provable global linear rate of convergence for a class of iterative parameter mixing methods based on functional approximation?

Key findings

• FADL achieves global linear rate of convergence (glrc), ensuring that the error decreases by a constant factor per iteration, leading to O(log(1/ε)) iterations to reach ε-optimality.
• FADL reduces the number of communication passes by up to several orders of magnitude compared to Terascale, especially in communication-heavy settings.
• Despite higher per-iteration computation, FADL achieves 1–10x speedup over Terascale across multiple datasets, including kdd2010, url, and webspam.
• The total computation-to-communication cost ratio for FADL is 1.2–1.6 on average, significantly better than CoCoA (0.1–0.16) and ADMM (1.3–3.5), indicating better balance.
• FADL outperforms CoCoA and ADMM in total training time even on medium- and low-dimensional datasets, demonstrating robustness across settings.
• The method supports flexible data usage, including overlapping feature partitions and resampling, without compromising convergence or performance.