[Paper Review] Unified Rules of Renewable Weighted Sums for Various Online Updating Estimations

Lu Lin, Weiyu Li+1 more|arXiv (Cornell University)|Aug 20, 2020

Advanced Statistical Process Monitoring1 references8 citations

TL;DR

This paper introduces a unified framework called Renewable Weighted Sums (RWS) for online updating estimation in streaming data environments. By leveraging cumulative weighted updates with adaptive distance and loss functions, RWS enables consistent, asymptotically normal, and oracle-efficient estimators without constraints on data batch count, applicable to nonparametric, parametric, and semiparametric models with minimal computational overhead.

ABSTRACT

This paper establishes unified frameworks of renewable weighted sums (RWS) for various online updating estimations in the models with streaming data sets. The newly defined RWS lays the foundation of online updating likelihood, online updating loss function, online updating estimating equation and so on. The idea of RWS is intuitive and heuristic, and the algorithm is computationally simple. This paper chooses nonparametric model as an exemplary setting. The RWS applies to various types of nonparametric estimators, which include but are not limited to nonparametric likelihood, quasi-likelihood and least squares. Furthermore, the method and the theory can be extended into the models with both parameter and nonparametric function. The estimation consistency and asymptotic normality of the proposed renewable estimator are established, and the oracle property is obtained. Moreover, these properties are always satisfied, without any constraint on the number of data batches, which means that the new method is adaptive to the situation where streaming data sets arrive perpetually. The behavior of the method is further illustrated by various numerical examples from simulation experiments and real data analysis.

Motivation & Objective

• To establish a unified theoretical framework for online updating estimation in streaming data models, addressing the lack of generalizable rules for likelihood, loss function, and estimating equations.
• To resolve the critical limitation in existing methods that require restrictive constraints like $k = O(n^c)$ on the number of data batches for statistical consistency.
• To develop a computationally simple, intuitive, and adaptive method applicable across nonparametric, parametric, and semiparametric models.
• To ensure estimation consistency, asymptotic normality, and oracle property under general conditions, even with perpetual data streaming.
• To demonstrate the method’s effectiveness through simulation and real data analysis, showing performance close to oracle estimators.

Proposed method

• Proposes a renewable weighted sum (RWS) framework defined by $\widehat{\theta}_k = \min_\theta \sum_{j=1}^{k-1} \sum_{i \in \mathbf{i}_j} J(\widehat{\theta}_j; Z_i) \|\theta - \widehat{\theta}_{k-1}\|^2 + L(\theta; \mathbf{d}_k)$, where $J$ is a weight function and $L$ is a loss function.
• Uses a recursive update rule that depends only on the previous estimator $\widehat{\theta}_{k-1}$, current data $\mathbf{d}_k$, and accumulated weights from prior batches, avoiding storage of raw historical data.
• Establishes equivalence between RWS and online updating estimating equations when $U_0(u) = u$, leading to $\sum_{j=1}^{k-1} \sum_{i \in \mathbf{i}_j} J(\widehat{\theta}_j; Z_i)(\theta - \widehat{\theta}_{k-1}) - U(\theta; \mathbf{d}_k) = 0$.
• Applies the RWS framework to nonparametric kernel regression, deriving an online bandwidth selection rule that adapts to streaming data and achieves optimal performance.
• Employs distance-based regularization via $\|\theta - \widehat{\theta}_{k-1}\|^2$ to stabilize updates and ensure convergence without requiring full data reprocessing.
• Extends the method to semiparametric models by combining parametric and nonparametric estimation components within the RWS structure.

Experimental results

Research questions

• RQ1 Can a unified framework be developed for online updating estimation that generalizes likelihood, loss function, and estimating equation across diverse streaming data models?
• RQ2 Can the RWS framework achieve estimation consistency and asymptotic normality without imposing constraints on the number of data batches?
• RQ3 Does the RWS method maintain the oracle property in nonparametric and semiparametric models under streaming conditions?
• RQ4 How does the performance of the online RWS estimator compare to full-data estimators in terms of prediction accuracy and convergence?
• RQ5 Can the RWS framework support adaptive bandwidth selection in nonparametric kernel regression without prior knowledge of terminal data size?

Key findings

• The RWS estimator achieves estimation consistency and asymptotic normality under general regularity conditions, even as the number of data batches grows without bound.
• The method attains the oracle property, meaning its performance approaches that of an estimator with full knowledge of the entire data stream.
• In nonparametric kernel regression, the online RWS estimator with sequentially chosen bandwidth performs nearly identically to the full-data N-W estimator, with the ratio of prediction errors approaching 1 as sample size increases.
• The RWS-based prediction error ratio relative to the full-data estimator stabilizes near 1, indicating minimal performance loss despite online-only processing.
• Compared to a partial-data N-W estimator trained on first 250 batches, the RWS method shows a decreasing error ratio over time, indicating superior timeliness and accuracy in dynamic environments.
• The method is computationally efficient and requires no storage of raw historical data, relying only on the last estimator and cumulative weights.