[Paper Review] Distributed Optimization for Client-Server Architecture with Negative Gradient Weights

Shripad Gade, Nitin H. Vaidya|arXiv (Cornell University)|Aug 12, 2016

Privacy-Preserving Technologies in Data24 references3 citations

TL;DR

This paper proposes a synchronous distributed optimization algorithm for a client-server architecture with multiple parameter servers, using randomized negative gradient weights to enhance privacy while ensuring convergence. The method combines interleaved consensus and projected gradient descent, proving convergence for arbitrary time-varying topologies and achieving faster convergence with time-varying weights due to reduced oscillation around the optimum.

ABSTRACT

Availability of both massive datasets and computing resources have made machine learning and predictive analytics extremely pervasive. In this work we present a synchronous algorithm and architecture for distributed optimization motivated by privacy requirements posed by applications in machine learning. We present an algorithm for the recently proposed multi-parameter-server architecture. We consider a group of parameter servers that learn a model based on randomized gradients received from clients. Clients are computational entities with private datasets (inducing a private objective function), that evaluate and upload randomized gradients to the parameter servers. The parameter servers perform model updates based on received gradients and share the model parameters with other servers. We prove that the proposed algorithm can optimize the overall objective function for a very general architecture involving $C$ clients connected to $S$ parameter servers in an arbitrary time varying topology and the parameter servers forming a connected network.

Motivation & Objective

• To address privacy concerns in distributed machine learning where clients hold sensitive data and share only gradients.
• To develop a provably convergent optimization algorithm for a multi-parameter server architecture with arbitrary time-varying communication topologies.
• To enhance privacy by introducing random, potentially negative gradient weights during client uploads, preventing parameter servers from inferring individual client data.
• To demonstrate that the algorithm maintains correctness and convergence despite the use of negative weights and dynamic server communication patterns.
• To analyze the impact of server-server communication topology on convergence speed and stability.

Proposed method

• The algorithm uses an interleaved consensus and projected gradient descent approach, where parameter servers perform consensus updates using doubly stochastic weight matrices.
• Clients compute and upload randomized gradients with weights that can be negative, obscuring true gradient direction from parameter servers.
• The method employs a time-varying learning rate Δ and ensures convergence under standard conditions (sum and sum of squares constraints).
• The weight matrix used in consensus is doubly stochastic, satisfying both the strict limit condition (SLC) and bounded upper condition (BUC).
• The system is modeled as a network of C clients and S parameter servers, with arbitrary time-varying client-to-server and server-to-server topologies.
• Convergence is proven under the assumption that the sum of gradients converges to zero and the decision set 𝒳 is convex and compact.

Experimental results

Research questions

• RQ1 Can a distributed optimization algorithm with negative gradient weights converge to the optimal solution in a multi-parameter server architecture?
• RQ2 How does the choice of server-server communication topology affect convergence speed and stability?
• RQ3 Does the use of time-varying, randomized gradient weights enhance privacy without compromising convergence?
• RQ4 Can the algorithm maintain correctness and convergence when the learning rate is fixed and the topology is time-varying?
• RQ5 What is the impact of negative weights on the trajectory of iterates and their deviation from the average?

Key findings

• The algorithm converges to the optimal solution for any arbitrary time-varying topology of clients and parameter servers, provided the parameter server network remains connected.
• The use of time-varying gradient weights leads to faster convergence, as negative weights prevent prolonged deviation from the optimum.
• The complete graph topology among parameter servers yields the fastest convergence due to optimal information mixing.
• The star topology performs worst in terms of convergence speed and RMS error reduction, despite being a common design choice.
• The cyclic graph topology outperforms the linear (path) topology due to better connectivity and faster mixing.
• The algorithm maintains convergence even when gradient weights are randomly changed at each iteration, suggesting robustness to dynamic weight assignment.