[Paper Review] LocalNewton: Reducing Communication Bottleneck for Distributed Learning

Vipul Gupta, Avishek Ghosh+4 more|arXiv (Cornell University)|May 16, 2021

Stochastic Gradient Optimization Techniques47 references4 citations

TL;DR

This paper proposes LocalNewton, a distributed second-order optimization algorithm that reduces communication overhead by allowing workers to perform multiple local Newton steps before synchronizing with the master. It achieves under 60% of the communication rounds and less than 40% of the end-to-end training time compared to state-of-the-art methods on AWS Lambda, with an adaptive variant that dynamically reduces local iterations to refine convergence.

ABSTRACT

To address the communication bottleneck problem in distributed optimization within a master-worker framework, we propose LocalNewton, a distributed second-order algorithm with local averaging. In LocalNewton, the worker machines update their model in every iteration by finding a suitable second-order descent direction using only the data and model stored in their own local memory. We let the workers run multiple such iterations locally and communicate the models to the master node only once every few (say L) iterations. LocalNewton is highly practical since it requires only one hyperparameter, the number L of local iterations. We use novel matrix concentration-based techniques to obtain theoretical guarantees for LocalNewton, and we validate them with detailed empirical evaluation. To enhance practicability, we devise an adaptive scheme to choose L, and we show that this reduces the number of local iterations in worker machines between two model synchronizations as the training proceeds, successively refining the model quality at the master. Via extensive experiments using several real-world datasets with AWS Lambda workers and an AWS EC2 master, we show that LocalNewton requires fewer than 60% of the communication rounds (between master and workers) and less than 40% of the end-to-end running time, compared to state-of-the-art algorithms, to reach the same training~loss.

Motivation & Objective

• To address the communication bottleneck in distributed optimization, especially in high-latency environments like serverless computing.
• To design a practical second-order method that minimizes communication rounds without sacrificing convergence speed.
• To develop an adaptive scheme that dynamically adjusts local iteration counts based on training progress.
• To provide theoretical guarantees using matrix concentration techniques for convergence under local averaging.
• To demonstrate significant end-to-end training time and communication round reductions on real-world datasets.

Proposed method

• LocalNewton performs multiple local Newton steps on each worker using only local data and model parameters, reducing the need for frequent master-worker synchronization.
• The algorithm uses a fixed number $ L $ of local iterations per communication round, with $ L $ chosen as a single hyperparameter.
• Theoretical convergence is established via novel matrix concentration techniques, showing convergence to a small norm ball around the global minimum.
• An adaptive variant, Adaptive LocalNewton, reduces $ L $ over time based on observed loss changes, eventually switching to GIANT when $ L=1 $.
• Step-sizes are computed via backtracking line-search at each worker to ensure descent and stability.
• The method is evaluated on AWS Lambda with 100 workers, using real-world datasets including w8a, covtype, EPSILON, a9a, and ijcnn1.

Experimental results

Research questions

• RQ1 Can a second-order optimization method with local averaging significantly reduce communication rounds in distributed learning?
• RQ2 How does the choice of local iteration count $ L $ affect convergence and communication efficiency?
• RQ3 Can an adaptive strategy for $ L $ improve model quality and reduce training time in high-latency environments?
• RQ4 Does LocalNewton achieve better end-to-end performance than state-of-the-art first-order and second-order methods in serverless settings?
• RQ5 What is the theoretical convergence behavior of LocalNewton under local averaging and limited communication?

Key findings

• Adaptive LocalNewton reduces communication rounds to less than 60% of those required by state-of-the-art algorithms to reach the same training loss on AWS Lambda.
• The end-to-end training time of LocalNewton is reduced to less than 40% of the baseline methods on AWS Lambda, directly due to fewer communication rounds.
• For all tested datasets, Adaptive LocalNewton gradually reduces $ L $ from an initial value of 3, eventually switching to GIANT when $ L=1 $.
• The algorithm achieves faster convergence by leveraging second-order information locally, enabling rapid progress toward the optimal solution with minimal communication.
• Empirical results show consistent improvements across five real-world datasets, including w8a, covtype, EPSILON, a9a, and ijcnn1, under identical worker counts and regularization.
• Theoretical analysis confirms that LocalNewton converges to a norm ball of small radius around the global minimum, validating its role as an effective initialization strategy.