[Paper Review] Stochastic Gradient Descent for Linear Systems with Missing Data

Anna Ma, Deanna Needell|arXiv (Cornell University)|Feb 23, 2017

Sparse and Compressive Sensing Techniques21 references4 citations

TL;DR

This paper proposes the mSGD algorithm, a stochastic gradient descent method tailored for solving large-scale linear systems with missing data. By adapting SGD to handle incomplete matrices through unbiased gradient estimates and projection steps, the method achieves linear convergence with fixed step sizes even for inconsistent systems, offering theoretical guarantees and empirical validation on simulated and real-world data.

ABSTRACT

Traditional methods for solving linear systems have quickly become impractical due to an increase in the size of available data. Utilizing massive amounts of data is further complicated when the data is incomplete or has missing entries. In this work, we address the obstacles presented when working with large data and incomplete data simultaneously. In particular, we propose to adapt the Stochastic Gradient Descent method to address missing data in linear systems. Our proposed algorithm, the Stochastic Gradient Descent for Missing Data method (mSGD), is introduced and theoretical convergence guarantees are provided. In addition, we include numerical experiments on simulated and real world data that demonstrate the usefulness of our method.

Motivation & Objective

• Address the challenge of solving large-scale linear systems when data is incomplete or contains missing entries.
• Develop a low-memory, iterative method suitable for massive datasets with missing values.
• Provide theoretical convergence guarantees for stochastic gradient descent in the presence of missing data.
• Demonstrate the effectiveness of the proposed method on both simulated and real-world datasets.

Proposed method

• Propose mSGD, a stochastic gradient descent variant that uses unbiased gradient estimates from randomly selected rows of the incomplete matrix A.
• Define the objective function as the least-squares minimization of ||Ax - b||²/2m, where A is partially observed.
• Use a projection step onto a convex set W to ensure iterates remain within a feasible domain.
• Incorporate a fixed step size strategy, enabling linear convergence even for inconsistent systems.
• Leverage the fact that ∇F(x⋆) = 0 holds in inconsistent cases due to residual orthogonality, preserving convergence properties.
• Apply Lipschitz continuity and strong convexity assumptions to derive convergence bounds.

Experimental results

Research questions

• RQ1 Can stochastic gradient descent be effectively adapted to solve linear systems when data is missing?
• RQ2 Does the proposed mSGD method achieve linear convergence with fixed step sizes for inconsistent linear systems?
• RQ3 How does mSGD perform in practice compared to standard methods on incomplete large-scale data?
• RQ4 What theoretical guarantees can be established for mSGD under missing data assumptions?

Key findings

• The mSGD algorithm achieves linear convergence with a fixed step size even for inconsistent linear systems, defying the typical need for diminishing step sizes.
• Theoretical analysis shows that the expected error decays geometrically, bounded by r^k ||x₀ - x⋆||² + α²G⋆/(1 - r), where r < 1.
• The method maintains convergence when ∇F(x⋆) = 0 holds due to residual orthogonality, even when Ax⋆ ≠ b.
• Numerical experiments on simulated and real-world data confirm the practical utility and robustness of mSGD.
• The convergence rate depends on the strong convexity parameter μ and the Lipschitz constant Lg of the gradient.