[Paper Review] Count-Min Tree Sketch: Approximate counting for NLP

Guillaume Pitel, Geoffroy Fouquier+2 more|arXiv (Cornell University)|Apr 19, 2016

Advanced Database Systems and Queries10 references3 citations

TL;DR

This paper proposes the Count-Min Tree Sketch (CMTS), a novel approximate counting data structure that leverages the Zipfian distribution of text data by combining pyramidal counters and barrier bits to significantly reduce relative error. CMTS achieves up to 100× lower average relative error (ARE) than standard Count-Min Sketch at perfect storage size, with 800% size improvement for equivalent error, outperforming logarithmic variants in both count and PMI estimation accuracy.

ABSTRACT

The Count-Min Sketch is a widely adopted structure for approximate event counting in large scale processing. In a previous work we improved the original version of the Count-Min-Sketch (CMS) with conservative update using approximate counters instead of linear counters. These structures are computationaly efficient and improve the average relative error (ARE) of a CMS at constant memory footprint. These improvements are well suited for NLP tasks, in which one is interested by the low-frequency items. However, if Log counters allow to improve ARE, they produce a residual error due to the approximation. In this paper, we propose the Count-Min Tree Sketch (Copyright 2016 eXenSa. All rights reserved) variant with pyramidal counters, which are focused toward taking advantage of the Zipfian distribution of text data.

Motivation & Objective

• To address the high relative error in approximate counting for low-frequency items in NLP, where standard Count-Min Sketch (CMS) performs poorly despite good absolute error bounds.
• To improve upon existing approximate counters like Count-Min-Log by exploiting the inherent Zipfian distribution of word frequencies in text data.
• To design a memory-efficient sketch structure that maintains high precision for low-frequency events while remaining computationally competitive with native dictionary implementations.
• To evaluate the performance of the proposed CMTS against CMS and Count-Min-Log variants in terms of relative error, absolute error, and PMI estimation accuracy on real NLP workloads.

Proposed method

• CMTS uses a hierarchical tree structure with counting bits and barrier bits, where barrier bits mark the start of higher-precision counting layers.
• The value of a counter is computed as v = c + 2×(2^b − 1), where b is the number of contiguous high-order barrier bits set to 1, and c is the value of the following b+1 counting bits.
• Incrementing a counter involves recalculating the new barrier count nb = min(layers, lsb((nv+2)/4)) and setting the corresponding bits in the counter structure.
• The structure supports efficient merging and querying via modified get and set algorithms that handle shared bits and potential overflow in multi-level conflict scenarios.
• The sketch uses a 128-bit base and 32-bit spire, with barrier bits being immutable once set to 1, enabling logarithmic-like growth with better precision control.
• The method is integrated with conservative update (CU) to reduce overcounting, and evaluated under high-pressure memory constraints.

Experimental results

Research questions

• RQ1 Can a tree-based approximate counting structure exploit the Zipfian distribution of text data to reduce relative error for low-frequency items compared to standard Count-Min Sketch?
• RQ2 How does the Count-Min Tree Sketch compare to Count-Min-Log in terms of average relative error (ARE) and root mean square error (RMSE) on unigram and bigram counts?
• RQ3 To what extent does CMTS improve the accuracy of downstream NLP tasks like Pointwise Mutual Information (PMI) estimation under memory constraints?
• RQ4 How does the computational efficiency of CMTS compare to native hash maps and standard CMS, especially in multithreaded environments?

Key findings

• At the ideal perfect storage size (100%), CMTS achieves an average relative error (ARE) of 10^-3, which is 100× lower than the standard Count-Min Sketch.
• CMTS reduces ARE by a factor of 100 compared to CMS at perfect storage size, with a size improvement ratio of approximately 800% for equivalent error levels.
• For PMI estimation, CMTS outperforms CMS by a factor of 10 in RMSE at the perfect storage size, and maintains superior performance even under high memory pressure.
• While CMTS degrades faster than other variants under extreme memory pressure (below 10% of ideal size), the resulting errors (RMSE ~2.5, ARE ~31) are too high for practical use, indicating the method remains effective in realistic memory regimes.
• CMTS remains computationally competitive with native C++ unordered_map structures and shows minimal precision degradation in unsynchronized multithreaded settings, even at very low error levels (10^-5).
• The Count-Min-Log variants (CMLS16-CU and CMLS8-CU) show diminishing returns beyond 200% of ideal size due to residual approximation errors, while CMTS continues to improve with increased memory allocation.