[Paper Review] Preconditioning in Expectation

Michael B. Cohen, Rasmus Kyng+3 more|arXiv (Cornell University)|Jan 24, 2014

Stochastic Gradient Optimization Techniques21 references3 citations

TL;DR

This paper introduces a novel preconditioning method that constructs ultra-sparsifiers through random sampling, achieving nearly-optimal performance in expectation without requiring exact spectral approximation. By combining this with recursive preconditioning and improved embedding techniques, the authors develop an SDD linear system solver running in expected time close to $m\log^{1/2}n$, significantly improving prior bounds.

ABSTRACT

We show that preconditioners constructed by random sampling can perform well without meeting the standard requirements of iterative methods. When applied to graph Laplacians, this leads to ultra-sparsifiers that in expectation behave as the nearly-optimal ones given by [Kolla-Makarychev-Saberi-Teng STOC`10]. Combining this with the recursive preconditioning framework by [Spielman-Teng STOC`04] and improved embedding algorithms, this leads to algorithms that solve symmetric diagonally dominant linear systems and electrical flow problems in expected time close to $m\log^{1/2}n$ .

Motivation & Objective

• To address the open problem of efficiently constructing nearly-optimal spectral and ultra-sparsifiers in nearly-linear time.
• To design a randomized preconditioning framework that performs well in expectation, even when standard iterative method requirements are not met.
• To achieve faster solvers for symmetric diagonally dominant (SDD) linear systems by leveraging random subgraph sampling and recursive preconditioning.
• To provide a constructive algorithmic answer to Batson et al.'s open question on efficient nearly-optimal sparsifier generation.

Proposed method

• The method uses random sampling of subgraphs (not just edges) to build preconditioners that, in expectation, achieve the same error reduction per iteration as optimal ultra-sparsifiers.
• It applies a randomized Richardson iteration with preconditioned Chebyshev iteration, where the preconditioner is constructed via random subgraph sampling with stretch and off-tree edge bounds.
• The algorithm integrates a recursive preconditioning framework inspired by Koutis et al., using a solver chain of progressively sparser graphs.
• It employs embedding techniques to transfer solutions and error guarantees between graphs and their tree approximations, ensuring stability and convergence.
• The analysis focuses on the intended application—iterative linear system solving—rather than requiring the sampled matrix to be a strong spectral approximation.
• It uses statistical bounds on error reduction in expectation, allowing for robust convergence even with large absolute errors in intermediate steps.

Experimental results

Research questions

• RQ1 Can random sampling of subgraphs yield preconditioners that perform as well as nearly-optimal ultra-sparsifiers in expectation?
• RQ2 Is it possible to construct nearly-optimal sparsifiers in nearly-linear time, resolving an open problem posed by Batson et al.?
• RQ3 Can the recursive preconditioning framework be adapted to work with randomized preconditioners that do not satisfy standard spectral approximation guarantees?
• RQ4 How can the error reduction per iteration be maintained in expectation when using random subgraph sampling instead of deterministic constructions?
• RQ5 Can the overall solver runtime be reduced to near $m\log^{1/2}n$ by combining random preconditioning with improved embedding and recursion?

Key findings

• The proposed algorithm solves symmetric diagonally dominant linear systems in expected time $\mathcal{O}(m\log^{1/2}n\log\log^{3+\delta}n\log(1/\epsilon))$ for any constant $\delta > 0$, achieving near-$m\log^{1/2}n$ complexity.
• The method constructs preconditioners via random sampling of subgraphs that, in expectation, match the error reduction of nearly-optimal ultra-sparsifiers from Kolla et al. (2010).
• The solver framework achieves relative error $\epsilon$ in $\mathcal{O}(m\log^{1/2}n\log\log^{3+\delta}n\log(1/\epsilon))$ expected time, with high probability.
• The algorithm is robust to large absolute errors in intermediate steps, relying on preconditioned Chebyshev iteration with strong convergence guarantees.
• The analysis shows that the recursive preconditioning framework remains stable under random sampling, even when the preconditioner does not exactly satisfy spectral approximation bounds.
• The approach enables the generation of electrical flows with approximate minimum energy in similar time, demonstrating broad applicability to dual problems in graph algorithms.