[Paper Review] Online Covariance Matrix Estimation in Stochastic Gradient Descent

Wanrong Zhu, Xi Chen+1 more|arXiv (Cornell University)|Feb 10, 2020

Stochastic Gradient Optimization Techniques42 references4 citations

TL;DR

This paper proposes a fully online, recursive estimator for the covariance matrix of averaged stochastic gradient descent (ASGD) iterates, enabling real-time statistical inference without storing past data. The method achieves consistency and optimal convergence rates comparable to offline estimators, allowing dynamic update of confidence intervals as new data arrives.

ABSTRACT

The stochastic gradient descent (SGD) algorithm is widely used for parameter estimation, especially for huge data sets and online learning. While this recursive algorithm is popular for computation and memory efficiency, quantifying variability and randomness of the solutions has been rarely studied. This paper aims at conducting statistical inference of SGD-based estimates in an online setting. In particular, we propose a fully online estimator for the covariance matrix of averaged SGD iterates (ASGD) only using the iterates from SGD. We formally establish our online estimator's consistency and show that the convergence rate is comparable to offline counterparts. Based on the classic asymptotic normality results of ASGD, we construct asymptotically valid confidence intervals for model parameters. Upon receiving new observations, we can quickly update the covariance matrix estimate and the confidence intervals. This approach fits in an online setting and takes full advantage of SGD: efficiency in computation and memory.

Motivation & Objective

• Address the lack of uncertainty quantification in online stochastic gradient descent (SGD) despite its computational and memory efficiency.
• Develop a fully online method to estimate the covariance matrix of ASGD iterates using only the SGD iterates, without requiring offline data storage.
• Enable real-time statistical inference by supporting dynamic updates of confidence intervals as new observations arrive.
• Ensure the estimator's convergence rate matches that of offline counterparts, maintaining statistical efficiency in online settings.
• Formally establish consistency and asymptotic normality for the proposed online covariance estimator under standard regularity conditions.

Proposed method

• Propose a recursive, online estimator for the covariance matrix of averaged SGD (ASGD) iterates using only the sequence of SGD iterates.
• Leverage the asymptotic normality of ASGD to construct confidence intervals based on the estimated covariance matrix.
• Design a block-based estimator that partitions SGD iterates into blocks to reduce variance and improve convergence properties.
• Use a fixed-width sequential stopping rule to determine termination in online inference, ensuring desired precision with high probability.
• Apply theoretical bounds on bias and variance of the estimator to prove consistency and convergence rates under regularity conditions.
• Establish that the mean squared error of the estimator decays at a rate comparable to offline methods, with optimal scaling in high-dimensional settings.

Experimental results

Research questions

• RQ1 Can a fully online estimator for the covariance matrix of ASGD iterates be constructed that maintains statistical efficiency and consistency?
• RQ2 What is the convergence rate of the proposed online covariance estimator compared to offline counterparts?
• RQ3 How can confidence intervals for model parameters be dynamically updated in real time using only SGD iterates?
• RQ4 What stopping rule ensures the desired precision in online inference while minimizing computational cost?
• RQ5 How does the block-based structure of the estimator affect bias and variance in finite samples?

Key findings

• The proposed online covariance estimator achieves consistency with a mean squared error that decays at a rate of $ O(n^{-1/eta} + n^{2eta heta - 2}) $, where $ eta $ controls block size and $ heta $ relates to step size decay.
• The convergence rate of the estimator matches that of offline methods, with the bias term decaying as $ O(n^{eta heta - 1}) $ and variance as $ O(n^{-1/eta}) $.
• The estimator enables dynamic, recursive updates of confidence intervals as new data arrives, supporting real-time statistical inference.
• Simulation results show that the fixed-width sequential stopping rule terminates within reasonable iterations (e.g., ~450K for $ d=20 $ logistic regression) with coverage probabilities near nominal levels (e.g., 0.84–0.91).
• The bias of the estimator is dominated by cross-block covariances, which decay exponentially with block separation, justifying the block-based design.
• Theoretical analysis confirms that the estimator’s bias and variance are both controlled under standard regularity conditions, ensuring asymptotic validity of inference.