[Paper Review] Moment-Based Quantile Sketches for Efficient High Cardinality Aggregation Queries

Edward Gan, Jialin Ding+3 more|arXiv (Cornell University)|Mar 6, 2018

Data Management and Algorithms70 references3 citations

TL;DR

This paper introduces the moments sketch, a compact quantile sketch that tracks sample and log-moments to enable fast, accurate quantile estimation with under 200 bytes of memory and sub-50ns merge times. By combining the method of moments and maximum entropy principle, it achieves <1% quantile error and accelerates query performance by up to 60× in Druid and 7.9× in MacroBase compared to existing sketches.

ABSTRACT

Interactive analytics increasingly involves querying for quantiles over sub-populations of high cardinality datasets. Data processing engines such as Druid and Spark use mergeable summaries to estimate quantiles, but summary merge times can be a bottleneck during aggregation. We show how a compact and efficiently mergeable quantile sketch can support aggregation workloads. This data structure, which we refer to as the moments sketch, operates with a small memory footprint (200 bytes) and computationally efficient (50ns) merges by tracking only a set of summary statistics, notably the sample moments. We demonstrate how we can efficiently and practically estimate quantiles using the method of moments and the maximum entropy principle, and show how the use of a cascade further improves query time for threshold predicates. Empirical evaluation on real-world datasets shows that the moments sketch can achieve less than 1 percent error with 15 times less merge overhead than comparable summaries, improving end query time in the MacroBase engine by up to 7 times and the Druid engine by up to 60 times.

Motivation & Objective

• To address the performance bottleneck of merging large numbers of quantile summaries in high-cardinality OLAP systems like Druid and Spark.
• To reduce memory and computational overhead in interactive analytics workloads requiring approximate quantile estimation across millions of dimension-value combinations.
• To design a summary data structure that supports efficient merge operations while maintaining high accuracy for quantile queries.
• To enable interactive query latencies in real-time monitoring and exploratory data analysis by minimizing summary merge time.
• To develop a cascaded filtering strategy that accelerates threshold-based quantile queries using progressively more accurate estimators.

Proposed method

• The moments sketch stores sample moments μi = (1/n)Σxi and log-moments νi = (1/n)Σlogi(x) to summarize data distributions compactly.
• Quantile estimation is performed using the method of moments and the principle of maximum entropy to infer the most unbiased distribution matching the stored moments.
• The sketch supports efficient merge operations via closed-form aggregation of moments, enabling sub-50ns merge times with minimal memory footprint.
• A cascade of estimators is introduced: starting with fast, approximate checks (e.g., range-based), progressing to increasingly accurate moment-based estimates, reducing the number of expensive full estimations.
• The system uses turnstile semantics to support sliding window queries by dynamically updating sketches through subtraction and merging.
• The approach is evaluated in production-like workloads using real-world datasets (e.g., Milan internet usage, Microsoft telemetry) across Druid and MacroBase engines.

Experimental results

Research questions

• RQ1 Can a summary data structure that tracks only moments achieve high-accuracy quantile estimation with minimal memory and computational cost?
• RQ2 How does the merge time of moment-based sketches compare to existing mergeable quantile summaries like GK-sketch or Merge12 in high-cardinality aggregation workloads?
• RQ3 To what extent can cascaded filtering strategies reduce query latency in threshold-based quantile queries?
• RQ4 Can the moments sketch maintain <1% quantile error across diverse real-world datasets while enabling interactive query performance?
• RQ5 How effective is the combination of moment-based estimation and maximum entropy in producing unbiased quantile estimates from limited summary statistics?

Key findings

• The moments sketch achieves less than 1% quantile error on real-world datasets while using only 200 bytes of memory and sub-50ns merge times.
• In the MacroBase engine, queries using the moments sketch with cascades are 7.9× faster than using the Merge12 sketch and 3.7× faster than a direct-estimate baseline.
• In the Druid engine, the moments sketch reduces query time by up to 60× compared to the Merge12 sketch due to significantly faster merge operations.
• The cascade mechanism reduces estimation time by over 250× compared to a naive baseline, with each stage processing fewer queries and improving throughput.
• Sliding window queries using the moments sketch with turnstile updates and cascades run 13× faster than using the Merge12 sketch.
• The method of moments combined with maximum entropy provides more accurate quantile estimates than alternative approaches on continuous real-valued data.