[Paper Review] Fast and Sample-Efficient Federated Low Rank Matrix Recovery from column-wise Linear and Quadratic Projections

Seyedehsara, Nayer+1 more|arXiv (Cornell University)|Feb 20, 2021

Sparse and Compressive Sensing Techniques4 citations

TL;DR

This paper proposes AltGD-Min, a fast, sample-efficient, and communication-efficient gradient descent algorithm for federated low-rank matrix recovery from column-wise linear and quadratic projections. It achieves $ ilde{O}((n+q)r^2/log(1/\\/epsilon)$ sample complexity and $ ilde{O}(m q n r \\/log(1/\\epsilon)$ time complexity under Gaussian sketching matrices, outperforming prior methods in speed and sample efficiency while enabling exponential error decay and federated deployment with $nr$ communication cost per node.

ABSTRACT

We study the following lesser-known low rank (LR) recovery problem: recover an $n imes q$ rank-$r$ matrix, $X^* =[x^*_1 , x^*_2, ..., x^*_q]$, with $r \ll \min(n,q)$, from $m$ independent linear projections of each of its $q$ columns, i.e., from $y_k := A_k x^*_k , k \in [q]$, when $y_k$ is an $m$-length vector with $m < n$. The matrices $A_k$ are known and mutually independent for different $k$. We introduce a novel gradient descent (GD) based solution called AltGD-Min. We show that, if the $A_k$s are i.i.d. with i.i.d. Gaussian entries, and if the right singular vectors of $X^*$ satisfy the incoherence assumption, then $ε$-accurate recovery of $X^*$ is possible with order $(n+q) r^2 \log(1/ε)$ total samples and order $ mq nr \log (1/ε)$ time. Compared with existing work, this is the fastest solution. For $ε< r^{1/4}$, it also has the best sample complexity. A simple extension of AltGD-Min also provably solves LR Phase Retrieval, which is a magnitude-only generalization of the above problem. AltGD-Min factorizes the unknown $X$ as $X = UB$ where $U$ and $B$ are matrices with $r$ columns and rows respectively. It alternates between a (projected) GD step for updating $U$, and a minimization step for updating $B$. Its each iteration is as fast as that of regular projected GD because the minimization over $B$ decouples column-wise. At the same time, we can prove exponential error decay for it, which we are unable to for projected GD. Finally, it can also be efficiently federated with a communication cost of only $nr$ per node, instead of $nq$ for projected GD.

Motivation & Objective

• To address the under-explored problem of low-rank matrix recovery from independent linear projections of each column, termed Low Rank column-wise Compressive Sensing (LRcCS).
• To develop a provably correct, fast, and communication-efficient algorithm for LRcCS in a federated setting with minimal data exchange per node.
• To achieve superior sample and time complexity compared to existing convex relaxation and alternating minimization methods.
• To extend the method to solve LR Phase Retrieval, a magnitude-only variant of LRcCS, with theoretical guarantees.
• To enable exponential error decay in recovery via a novel alternating minimization scheme with decoupled column-wise updates.

Proposed method

• AltGD-Min factorizes the unknown matrix $\bm{X}^*$ as $\bm{X}^* = \bm{U} \bm{B}$, where $\bm{U}$ is an $n \times r$ orthonormal basis matrix and $\bm{B}$ is an $r \times q$ coefficient matrix.
• It alternates between a projected gradient descent step for updating $\bm{U}$ and a closed-form minimization step for updating $\bm{B}$, which decouples column-wise and enables fast computation.
• The algorithm uses a subspace distance (SD) measure $\mathrm{SD}(\bm{U}_1, \bm{U}_2) = \| (\bm{I} - \bm{U}_1\bm{U}_1^\top)\bm{U}_2 \|_F$ to track convergence and subspace alignment.
• The method is extended to LR Phase Retrieval by replacing linear projections with magnitude-only measurements $\bm{y}_{(\text{mag})} = |\bm{A}_k \bm{x}_k^*|$, maintaining provable recovery guarantees.
• Communication efficiency is achieved by transmitting only $nr$ parameters per node per round, instead of $nq$, enabling scalable federated deployment.
• Theoretical analysis relies on incoherence assumptions on the right singular vectors of $\bm{X}^*$ and i.i.d. Gaussian sketching matrices $\bm{A}_k$.

Experimental results

Research questions

• RQ1 Can a gradient-based method achieve faster convergence and lower sample complexity than existing convex relaxation and alternating minimization approaches for LRcCS?
• RQ2 Does the proposed AltGD-Min algorithm enable exponential error decay in low-rank matrix recovery from column-wise projections?
• RQ3 Can AltGD-Min be efficiently federated with $O(nr)$ communication cost per node, significantly reducing bandwidth compared to standard methods?
• RQ4 Does the algorithm maintain provable recovery guarantees under the same assumptions as prior work, but with improved time and sample complexity?
• RQ5 Can the method be extended to solve LR Phase Retrieval with similar theoretical guarantees and efficiency?

Key findings

• AltGD-Min achieves $\epsilon$-accurate recovery of an $n \times q$ rank-$r$ matrix with $\tilde{O}((n+q)r^2 \log(1/\epsilon))$ total samples, which is optimal for $\epsilon < r^{1/4}$.
• The algorithm runs in $\tilde{O}(m q n r \log(1/\epsilon))$ time, making it the fastest known method for LRcCS under i.i.d. Gaussian sketching matrices.
• The method ensures exponential error decay, a property not provable for standard projected gradient descent.
• Communication cost is reduced to $nr$ per node per round in federated settings, significantly improving scalability over methods requiring $nq$ transmission.
• The algorithm is extended to LR Phase Retrieval and maintains provable recovery guarantees under the same assumptions.
• Theoretical analysis confirms that the method works under standard incoherence and Gaussian sketching assumptions, with high probability bounds derived via sub-Gaussian concentration and epsilon-net arguments.