[Paper Review] Distributed Averaging Methods for Randomized Second Order Optimization

Burak Bartan, Mert Pilancı|arXiv (Cornell University)|Feb 16, 2020

Sparse and Compressive Sensing Techniques17 references4 citations

TL;DR

This paper proposes unbiased distributed averaging methods for randomized second-order optimization, using sketched Hessian approximations to reduce communication and computation costs. It provides closed-form formulas for regularization and step size that eliminate bias in sketched solutions, enabling convergence to the true optimum without additional cost, especially effective in heterogeneous and serverless computing environments.

ABSTRACT

We consider distributed optimization problems where forming the Hessian is computationally challenging and communication is a significant bottleneck. We develop unbiased parameter averaging methods for randomized second order optimization that employ sampling and sketching of the Hessian. Existing works do not take the bias of the estimators into consideration, which limits their application to massively parallel computation. We provide closed-form formulas for regularization parameters and step sizes that provably minimize the bias for sketched Newton directions. We also extend the framework of second order averaging methods to introduce an unbiased distributed optimization framework for heterogeneous computing systems with varying worker resources. Additionally, we demonstrate the implications of our theoretical findings via large scale experiments performed on a serverless computing platform.

Motivation & Objective

• To address the bias in averaged sketched solutions for randomized second-order optimization, which limits scalability in massively parallel systems.
• To develop a framework for unbiased parameter averaging in distributed settings with varying worker resources and sketching dimensions.
• To enable efficient, low-overhead optimization on resource-constrained, serverless platforms like AWS Lambda.
• To provide theoretical guarantees on bias correction and convergence for both regularized and unregularized problems.
• To demonstrate scalability and performance gains through large-scale experiments on a serverless computing platform.

Proposed method

• Uses sketching matrices (e.g., Gaussian, SJLT) to approximate the Hessian and reduce computational cost in second-order optimization.
• Derives closed-form formulas for regularization parameters and step sizes that eliminate bias in averaged sketched solutions.
• Introduces a distributed iterative Hessian sketch algorithm with bias-corrected averaging across heterogeneous workers.
• Applies the framework to both regularized least squares and general convex optimization problems with theoretical convergence guarantees.
• Employs a master-worker architecture where workers compute local sketched solutions and send them to a central master for averaging.
• Validates the method on AWS Lambda, reusing serverless functions across iterations to minimize setup overhead.

Experimental results

Research questions

• RQ1 How can bias be eliminated in averaged sketched solutions for second-order optimization when regularization is involved?
• RQ2 What closed-form formulas for regularization and step size ensure unbiased convergence in distributed randomized Hessian approximation?
• RQ3 How does the proposed method scale on heterogeneous computing systems with varying worker capabilities?
• RQ4 Can the method be efficiently deployed on serverless platforms with limited per-node resources and transient execution?
• RQ5 What is the impact of bias correction on convergence rate and final solution accuracy in large-scale optimization?

Key findings

• The paper derives explicit formulas for regularization parameters that eliminate bias in averaged sketched solutions, ensuring convergence to the true optimum.
• Bias correction requires adjusting the regularization coefficient in sub-problems based on sketch size and worker-specific problem dimensions.
• The method achieves unbiased estimation without additional computational cost, improving performance over naive averaging.
• Experiments on AWS Lambda show the algorithm scales effectively, with iterations completing in as little as 5 seconds on a 44 GB dataset.
• Gaussian and SJLT sketching matrices outperform other sketches, with SJLT offering lower computational cost and strong performance.
• The framework mitigates the straggler problem in distributed systems by avoiding reliance on slowest workers through efficient, low-communication design.