[Paper Review] Randomized Iterative Methods for Linear Systems: Momentum, Inexactness and Gossip

Nicolas Loizou|arXiv (Cornell University)|Sep 26, 2019

Stochastic Gradient Optimization Techniques195 references4 citations

TL;DR

This paper introduces randomized iterative methods with momentum, inexactness, and gossip-based communication for solving large-scale linear systems and optimization problems. It establishes the first provably accelerated linear convergence rates for randomized gossip algorithms and proposes novel privacy-preserving variants, significantly improving efficiency and scalability in distributed settings.

ABSTRACT

In the era of big data, one of the key challenges is the development of novel optimization algorithms that can accommodate vast amounts of data while at the same time satisfying constraints and limitations of the problem under study. The need to solve optimization problems is ubiquitous in essentially all quantitative areas of human endeavor, including industry and science. In the last decade there has been a surge in the demand from practitioners, in fields such as machine learning, computer vision, artificial intelligence, signal processing and data science, for new methods able to cope with these new large scale problems. In this thesis we are focusing on the design, complexity analysis and efficient implementations of such algorithms. In particular, we are interested in the development of randomized iterative methods for solving large scale linear systems, stochastic quadratic optimization problems, the best approximation problem and quadratic optimization problems. A large part of the thesis is also devoted to the development of efficient methods for obtaining average consensus on large scale networks.

Motivation & Objective

• To design efficient randomized iterative methods for solving large-scale linear systems and stochastic quadratic optimization problems.
• To extend first-order methods with heavy ball momentum and analyze their convergence in a unified framework for convex quadratic problems.
• To relax exact subproblem solution requirements by introducing inexact variants, reducing computational cost while maintaining convergence.
• To develop a general framework for randomized gossip algorithms that enables block, accelerated, and privacy-preserving variants for distributed average consensus.
• To protect private data in distributed networks by introducing novel privacy-preserving mechanisms in gossip protocols without sacrificing convergence guarantees.

Proposed method

• Introduces stochastic momentum as a computationally cheaper alternative to deterministic momentum, enabling faster iterations in sparse data regimes.
• Analyzes inexact variants of SGD, stochastic Newton, proximal point, and subspace ascent by allowing approximate solutions to subproblems with bounded error.
• Establishes a connection between randomized Kaczmarz-type methods and gossip algorithms via randomized projection methods for linear systems.
• Develops a unified framework for gossip algorithms by modeling them as randomized iterative solvers for consensus systems, enabling block and accelerated variants.
• Proposes three privacy-preserving gossip protocols: Binary Oracle, ε-Gap Oracle, and Controlled Noise Insertion, which weaken the information shared during updates.
• Uses duality to design algorithms that solve the dual of the best approximation problem, ensuring convergence to average consensus while preserving privacy.

Experimental results

Research questions

• RQ1 Can stochastic momentum be used to accelerate randomized iterative methods for linear systems and optimization, and does it achieve linear convergence?
• RQ2 What is the convergence rate of inexact variants of stochastic first-order methods when subproblems are solved approximately?
• RQ3 Can randomized gossip algorithms be designed with provably accelerated convergence rates for distributed average consensus?
• RQ4 How can privacy be preserved in gossip-based consensus algorithms without compromising convergence or accuracy?
• RQ5 Can the framework of randomized iterative methods for linear systems be extended to unify and generalize existing gossip protocols?

Key findings

• The paper establishes the first global non-asymptotic linear convergence rate for the stochastic heavy ball method in convex quadratic problems.
• Stochastic momentum methods achieve better overall complexity than deterministic momentum variants in sparse data regimes with small momentum parameters.
• Inexact variants of stochastic methods maintain linear convergence under bounded inexactness error, with significantly reduced time-to-solution despite more iterations.
• The proposed accelerated gossip protocols achieve provably faster linear convergence than existing randomized gossip algorithms, validated on wireless sensor network topologies.
• The privacy-preserving gossip protocols—Binary Oracle, ε-Gap Oracle, and Controlled Noise Insertion—achieve convergence to average consensus while protecting private node values.
• Theoretical iteration complexity bounds are provided for all proposed privacy-preserving protocols, and extensive numerical experiments confirm practical convergence on standard network topologies.