[Paper Review] Adapted Stochastic Gradient Descent for Linear Systems with Missing Data

Anna Ma, Deanna Needell|arXiv (Cornell University)|Feb 23, 2017

Stochastic Gradient Optimization Techniques21 references3 citations

TL;DR

This paper proposes mSGD, an adapted Stochastic Gradient Descent method for solving linear systems with missing data. By modifying the gradient update rule to account for incomplete entries, mSGD achieves theoretical convergence and demonstrates strong performance on both simulated and real-world datasets, offering a scalable solution for large-scale, incomplete linear systems.

ABSTRACT

Traditional methods for solving linear systems have quickly become impractical due to an increase in the size of available data. Utilizing massive amounts of data is further complicated when the data is incomplete or has missing entries. In this work, we address the obstacles presented when working with large data and incomplete data simultaneously. In particular, we propose to adapt the Stochastic Gradient Descent method to address missing data in linear systems. Our proposed algorithm, the Adapted Stochastic Gradient Descent for Missing Data method (mSGD), is introduced and theoretical convergence guarantees are provided. In addition, we include numerical experiments on simulated and real world data that demonstrate the usefulness of our method.

Motivation & Objective

• To address the challenge of solving large-scale linear systems when data is incomplete or contains missing entries.
• To develop a scalable optimization method that maintains efficiency and convergence despite missing data.
• To provide theoretical convergence guarantees for the proposed algorithm under realistic data conditions.
• To validate the method’s effectiveness through numerical experiments on both synthetic and real-world datasets.

Proposed method

• The mSGD algorithm modifies the standard stochastic gradient descent update rule to handle missing entries by ignoring them during gradient computation.
• It uses a randomized sampling strategy over the available data entries to iteratively update the solution vector.
• The method incorporates a step-size schedule that ensures convergence under mild assumptions on the data and system structure.
• Theoretical analysis establishes convergence in expectation to the true solution under conditions of bounded missingness and consistent sampling.
• The algorithm is designed to be memory-efficient and suitable for streaming or large-scale data applications.
• The approach treats missing entries as unobserved and adjusts the gradient contribution only over observed data points.

Experimental results

Research questions

• RQ1 Can stochastic gradient descent be effectively adapted to solve linear systems when data contains missing entries?
• RQ2 Does the proposed mSGD method maintain convergence to the true solution under missing data conditions?
• RQ3 How does mSGD compare to traditional methods in terms of scalability and accuracy on incomplete datasets?
• RQ4 What theoretical guarantees can be established for the convergence of mSGD in the presence of missing data?

Key findings

• The mSGD algorithm achieves convergence in expectation to the true solution of the linear system under mild assumptions on data availability and sampling.
• Numerical experiments show that mSGD outperforms standard SGD and other baseline methods in terms of solution accuracy on datasets with missing entries.
• The method scales efficiently to large-scale problems, maintaining low memory and computational overhead.
• Empirical results on real-world data confirm the robustness and practical utility of mSGD in incomplete data scenarios.
• Theoretical analysis confirms that mSGD converges even when data is missing at random, provided the sampling mechanism is consistent.