[Paper Review] COLA: Decentralized Linear Learning

Lie He, An Bian+1 more|arXiv (Cornell University)|Aug 13, 2018

Privacy-Preserving Technologies in Data43 citations

TL;DR

COLA is a fully decentralized framework for training generalized linear models on-device over arbitrary networks, providing convergence guarantees, communication efficiency, and resilience to topology changes and node availability.

ABSTRACT

Decentralized machine learning is a promising emerging paradigm in view of global challenges of data ownership and privacy. We consider learning of linear classification and regression models, in the setting where the training data is decentralized over many user devices, and the learning algorithm must run on-device, on an arbitrary communication network, without a central coordinator. We propose COLA, a new decentralized training algorithm with strong theoretical guarantees and superior practical performance. Our framework overcomes many limitations of existing methods, and achieves communication efficiency, scalability, elasticity as well as resilience to changes in data and participating devices.

Motivation & Objective

• Motivate decentralized learning to protect data ownership and privacy while enabling joint training on distributed devices.
• Develop a decentralized algorithm for generalized linear models with convergence guarantees.
• Ensure communication efficiency, scalability, and resilience to dynamic network topology and data changes.
• Leverage primal-dual techniques to bridge centralized CoCoA ideas to a fully decentralized setting.
• Provide practical diagnostics and certificates to monitor learning progress in decentralized deployments.

Proposed method

• Formulates the learning problem in primal and dual forms suitable for decentralized optimization (A) and (B).
• Introduces CoLa with local data-local subproblems solved approximately at each node (Theta-approximation).
• Uses local estimates v_k of the shared state and a gossip-like mixing matrix W to exchange information without a central coordinator.
• Defines the local subproblem G_k^sigma' that depends only on local data A_[k] and neighboring v_l terms (equation 2).
• Aggregates updates via v_k updates and a global aggregation parameter gamma to ensure progress.
• Establishes convergence guarantees: linear rate for strongly convex objectives and sublinear rate for general convex objectives, with data-dependent constants and local duality certificates.

Experimental results

Research questions

• RQ1 Can decentralized optimization achieve comparable convergence guarantees to centralized methods for generalized linear models without assuming i.i.d. data across devices?
• RQ2 How can we design a communication-efficient, elastic, and fault-tolerant decentralized algorithm that tolerates changing networks and data distributions?
• RQ3 What convergence rates (linear vs sublinear) can be proven under strong vs general convexity in a fully decentralized setting?
• RQ4 How can practitioners monitor progress locally without a central coordinator (local certificates)?

Key findings

• CoLa achieves linear convergence for strongly convex components and sublinear convergence for general convex objectives.
• The algorithm is resilient to topology changes, node dropouts, and data/partition changes, with no need for parameter tuning.
• CoLa outperforms decentralized baselines like DIGing and D-ADMM in communication rounds and time in empirical tests.
• Local certificates enable diagnosing progress without global synchronization or centralized aggregation.
• The method supports time-varying graphs and can leverage improved second-order subproblems for further gains.