[论文解读] Sublime: Sublinear Error & Space for Unbounded Skewed Streams

Navid Eslami, Ioana O. Bercea+2 more|arXiv (Cornell University)|Mar 15, 2026

Advanced Database Systems and Queries被引用 0

一句话总结

tldr: Sublime 是一个框架，通过使用可变长度计数器（VALE）和自适应增长，将频率估计草图推广到无界倾斜流，达到亚线性误差和空间，应用于 CMS、Count Sketch 和 Misra-Gries。

ABSTRACT

Modern stream processing systems often need to track the frequency of distinct keys in a data stream in real-time. Since maintaining exact counts can require a prohibitive amount of memory, many applications rely on compact, probabilistic data structures known as frequency estimation sketches to approximate them. However, mainstream frequency estimation sketches fall short in two critical aspects. First, they are memory-inefficient under skewed workloads because they use uniformly-sized counters to count the keys, thus wasting memory on storing the leading zeros of many small counts. Second, their estimation error deteriorates at least linearly with the length of the stream--which may grow indefinitely--because they rely on a fixed number of counters. We present Sublime, a framework that generalizes frequency estimation sketches to address these challenges. To reduce memory footprint under skew, Sublime begins with short counters and dynamically elongates them as they overflow, storing their extensions within the same cache line. It employs efficient bit manipulation routines to quickly locate and access a counter's extensions. To maintain accuracy as the stream grows, Sublime also expands its number of counters at a configurable rate, exposing a new spectrum of accuracy-memory tradeoffs that applications can tune to their needs. We apply Sublime to both Count-Min Sketch and Count Sketch. Through theoretical analysis and empirical evaluation, we show that Sublime significantly improves accuracy and memory over the state of the art while maintaining competitive or superior performance.

研究动机与目标

• Motivate the need for accurate frequency estimation under highly skewed data and unbounded stream growth.
• Introduce Sublime to address memory waste under skew and linear error growth with stream length.
• Provide a general encoding framework (VALE) and adaptive resizing to achieve sublinear error and space.
• Demonstrate Sublime’s applicability by applying it to Count-Min Sketch, Count Sketch, and Misra-Gries.
• Analyze theoretical bounds and validate performance through experiments.

提出的方法

• Introduce VALE: variable-length encoding that stores each counter as a Stub and extends it when overflow occurs within the same cache line.
• Partition counters into Chunks and track overflows with an Overflows Bitmap to locate extensions efficiently using rank/select operations.
• Store extensions as 2-bit fragments encoding base-3 digits, enabling compact representation of higher-order bits.
• Provide an external tails mechanism to handle extension pool exhaustion and maintain constant-time queries.
• Adaptively tune stub length and number of counters per chunk (c and s) to minimize memory footprint based on workload skew and growth.
• Apply Sublime to CMS (SublimeCMS), Count Sketch (SublimeCountSketch), and Misra-Gries, and prove a lower bound on space for expandable sketches.

实验结果

研究问题

• RQ1 How can frequency estimation sketches achieve sublinear memory usage under skewed workloads?
• RQ2 Can we maintain sublinear error growth with respect to stream length by dynamically expanding the sketch?
• RQ3 How can variable-length encoding (VALE) and fast extension lookup be implemented efficiently in practice?
• RQ4 What is the trade-off frontier between accuracy and memory, and how can tightening/loosening parameters affect it?
• RQ5 How does Sublime perform relative to state-of-the-art sketches across CMS, Count Sketch, and Misra-Gries in theory and practice?

主要发现

• Sublime reduces memory footprint under skew compared to CMS and skew-aware variants while maintaining or improving accuracy.
• Sublime achieves sublinear error growth with respect to stream length by expanding the number of counters over time.
• VALE enables constant-time access to variable-length counters by colocating stubs with extensions in the same cache line and using rank/select-based locating of extensions.
• The framework applies effectively to CMS, Count Sketch, and Misra-Gries, providing a Pareto frontier between accuracy and memory and improving estimation for join size tasks.
• A lower bound on minimum space for expandable sketches is established, with Sublime’s memory footprint closely matching this bound.