[Paper Review] Sketching the order of events

Terry Lyons, Harald Oberhauser|arXiv (Cornell University)|Aug 31, 2017

Data Visualization and Analytics12 references3 citations

TL;DR

This paper introduces a novel sketching framework for massive data streams that generalizes traditional stream sketches to 'ordered moments' of arbitrary order, enabling efficient, universal, and scalable feature representation. The method achieves perfect classification of Markovian stream patterns using second-order features, even when first-order sketches fail, by leveraging randomized linear projections with theoretical guarantees on error and memory efficiency.

ABSTRACT

We introduce features for massive data streams. These stream features can be thought of as "ordered moments" and generalize stream sketches from "moments of order one" to "ordered moments of arbitrary order". In analogy to classic moments, they have theoretical guarantees such as universality that are important for learning algorithms.

Motivation & Objective

• To address the challenge of efficiently summarizing massive, ordered data streams with minimal memory and computational cost.
• To generalize classic stream sketches (e.g., count-min) from first-order moments to higher-order ordered moments for richer feature representation.
• To provide theoretical guarantees on approximation error and universality, enabling use in downstream learning algorithms.
• To enable robust detection of temporal patterns in streams, such as Markovian regime shifts, where standard sketches fail.
• To design a streaming algorithm that supports real-time updates, logarithmic space complexity, and accurate pattern queries.

Proposed method

• Proposes a feature map Φ(σ) that encodes ordered moments of arbitrary order, where coordinates correspond to patterns (e.g., sequences of events) in the stream.
• Uses randomized linear projections via a structured random matrix L to compute a sketch  LΦ(σ), enabling low-dimensional approximation of Φ(σ) with high probability.
• Employs hash functions to map events to a smaller feature space, ensuring logarithmic space complexity in |A| and streaming efficiency.
• Derives theoretical bounds on relative error (1−ε) ≤ |Φi(σ)−Φ̂i(σ)| / ||Φ(σ)|| ≤ (1+ε) with probability >1−δ, ensuring robustness.
• Applies the sketch to non-commutative operations such as stream concatenation, which correspond to algebraic multiplication in feature space.
• Adapts the framework to handle turnstile and cash register stream models, with extensions to ℓ₂-error bounds via modifications analogous to classic sketching.

Experimental results

Research questions

• RQ1 Can higher-order ordered moments be efficiently sketched in a streaming setting while preserving theoretical error guarantees?
• RQ2 Can such sketches enable universal representation of non-linear functionals of streams, supporting standard learning algorithms?
• RQ3 Can the method distinguish subtle temporal patterns in streams—such as Markovian regime shifts—where first-order sketches fail?
• RQ4 How does the sketch perform in terms of memory efficiency and accuracy when distinguishing streams with similar heavy hitters but different temporal orderings?
• RQ5 What is the trade-off between sketch size (number of hash functions) and approximation accuracy in real-world stream scenarios?

Key findings

• The sketch achieves a memory compression ratio of up to 6012.50× (from 6 hours of ground-truth computation) when using 4 hash functions and |A|=100, with error ≈3923.47 for patterns up to length 3.
• With 32 hash functions and |A|=100, the memory ratio reaches 6012.50× while reducing error to 3923.47, demonstrating scalability and accuracy.
• For streams with 10⁵ events and |A|=10⁵, the method classifies Markovian regime shifts with 100% accuracy using second-order features (M=2), while first-order features (M=1) degrade to 57% accuracy as p and q approach each other.
• The sketch enables real-time classification in ~0.15 seconds per stream, with 20 hash functions and 10 target letters, outperforming baseline random guessing (50% accuracy) as p and q grow closer.
• The method maintains robustness under noisy observations and supports scaling limits, enabling analysis of large-scale stream behavior.
• The framework generalizes classic sketching (e.g., count-min) by extending it to higher-order moments, enabling algebraic operations like non-commutative multiplication for stream concatenation.