[Paper Review] Fast Moment Estimation in Data Streams in Optimal Space

Daniel M. Kane, Jelani Nelson+2 more|arXiv (Cornell University)|Jul 23, 2010

Advanced Database Systems and Queries31 references4 citations

TL;DR

This paper presents a space-optimal algorithm for estimating the $p$th frequency moment $F_p$ of a data stream with $0 < p < 2$, achieving a nearly exponential improvement in update time over prior space-optimal methods. By combining a novel hash-based heavy hitter detection with a refined variance-reduction technique using pairwise independent randomness, the algorithm attains $O(\log^2(1/\varepsilon)\log\log(1/\varepsilon))$ update time while maintaining optimal $O(\varepsilon^{-2}\log(mM) + \log\log n)$ space usage.

ABSTRACT

We give a space-optimal algorithm with update time O(log^2(1/eps)loglog(1/eps)) for (1+eps)-approximating the pth frequency moment, 0 &lt; p &lt; 2, of a length-n vector updated in a data stream. This provides a nearly exponential improvement in the update time complexity over the previous space-optimal algorithm of [Kane-Nelson-Woodruff, SODA 2010], which had update time Omega(1/eps^2).

Motivation & Objective

• To design a streaming algorithm that achieves optimal space complexity for estimating the $p$th frequency moment $F_p$ of a vector in a data stream, where $0 < p < 2$.
• To significantly reduce the update time of space-optimal algorithms for $F_p$ estimation, especially for $p \in (0,2)$, beyond prior known bounds.
• To achieve a nearly exponential improvement in update time complexity while preserving optimal space usage, making the algorithm practical for high-speed network and database applications.
• To develop a method that maintains high accuracy with constant success probability using only $O(\varepsilon^{-2}\log(mM) + \log\log n)$ bits of space and $O(\log^2(1/\varepsilon)\log\log(1/\varepsilon))$ update time.

Proposed method

• The algorithm uses a two-phase approach: first identifying heavy hitters via a hash function $h_1$ that maps coordinates to a smaller domain, enabling efficient detection of significant contributions to $F_p$.
• It employs a refined estimator, $\mathsf{LightEstimator}$, which computes the $p$th moment of the non-heavy components using a polynomial-based method with $O(\varepsilon^{-2})$ parallel repetitions to reduce variance.
• The method leverages pairwise independent randomness across multiple estimators to minimize space and update time, allowing the entire coefficient generation for degree-$O(1/\varepsilon^p)$ polynomials to be done with just two coefficient vectors.
• A heavy hitter detection module $\mathsf{F_pHH}$ identifies coordinates with $|x_w|^p \geq \varepsilon^2 \|x\|_p^p$, ensuring that only significant contributions are tracked with high accuracy.
• The algorithm combines estimates from the heavy hitter component ($\mathsf{HighEnd}$) and the light component ($\mathsf{LightEstimator}$) to produce a final $F_p$ estimate within $1 \pm \varepsilon$ of the true value.
• Success probability is amplified via median-of-medians of independent runs, ensuring $2/3$ success probability per run, which can be boosted to $1 - \delta$ using $O(\log(1/\delta))$ repetitions.

Experimental results

Research questions

• RQ1 Can we achieve space-optimality for $F_p$ estimation in data streams with $0 < p < 2$ while drastically reducing update time compared to prior space-optimal algorithms?
• RQ2 Is it possible to reduce the update time from $\Omega(\varepsilon^{-2})$ to $\tilde{O}(\log^2(1/\varepsilon)\log\log(1/\varepsilon))$ while maintaining optimal space complexity?
• RQ3 How can we design a streaming algorithm that uses only $O(\varepsilon^{-2}\log(mM) + \log\log n)$ bits but achieves near-constant update time for $p \in (0,2)$?
• RQ4 Can we maintain high accuracy in $F_p$ estimation using a combination of heavy hitter detection and variance-reduced polynomial estimators with minimal randomness?

Key findings

• The algorithm achieves space complexity of $O(\varepsilon^{-2}\log(mM) + \log\log n)$ bits, which is optimal for $F_p$ estimation with $0 < p < 2$.
• The update time is $O(\log^2(1/\varepsilon)\log\log(1/\varepsilon))$, representing a nearly exponential improvement over the previous space-optimal algorithm with $\Omega(\varepsilon^{-2})$ update time.
• The algorithm outputs a $(1 \pm \varepsilon)$-approximation to $F_p = \|x\|_p^p$ with probability at least $2/3$ using only $O(\varepsilon^{-2}\log(mM) + \log\log n)$ space.
• The method uses only pairwise independent randomness across estimators, enabling efficient coefficient generation with just two coefficient vectors, which keeps the update time low.
• The reporting time is $O(\varepsilon^{-2}\log^2(1/\varepsilon)\log\log(1/\varepsilon))$, which is acceptable for practical deployment in high-speed streaming environments.
• The success probability can be amplified to $1 - \delta$ by taking the median of $O(\log(1/\delta))$ independent runs, preserving the space and update time bounds.