[Paper Review] RobustBF: A High Accuracy and Memory Efficient 2D Bloom Filter

Nayak, Sabuzima, Patgiri, Ripon|arXiv (Cornell University)|Jun 6, 2021

Caching and Content Delivery16 references4 citations

TL;DR

This paper proposes robustBF, a novel 2D Bloom Filter that achieves near-zero false positive probability with up to 44× lower memory usage than counting Bloom Filters and 10× lower than standard Bloom Filters, by optimizing the Murmur hash function and using a prime-numbered 2D bit array structure to enhance accuracy and efficiency without sacrificing performance.

ABSTRACT

Bloom Filter is an important probabilistic data structure to reduce memory consumption for membership filters. It is applied in diverse domains such as Computer Networking, Network Security and Privacy, IoT, Edge Computing, Cloud Computing, Big Data, and Biometrics. But Bloom Filter has an issue of the false positive probability. To address this issue, we propose a novel robust Bloom Filter, robustBF for short. robustBF is a 2D Bloom Filter, capable of filtering millions of data with high accuracy without compromising the performance. Our proposed system is presented in two-fold. Firstly, we modify the murmur hash function, and test all modified hash functions for improvements and select the best-modified hash function experimentally. Secondly, we embed the modified hash functions in 2D Bloom Filter. Our experimental results show that robustBF is better than standard Bloom Filter and counting Bloom Filter in every aspect. robustBF exhibits nearly zero false positive probability with more than $10 imes$ and $44 imes$ lower memory consumption than standard Bloom filter and counting Bloom Filter, respectively.

Motivation & Objective

• To address the persistent issue of high false positive probability in traditional Bloom Filters while minimizing memory usage.
• To improve the accuracy and memory efficiency of membership filters without degrading insertion or lookup performance.
• To design a 2D Bloom Filter variant that reduces false positives to nearly zero under a target false positive rate of 0.001.
• To optimize the Murmur hash function for better distribution and lower collision rates in the 2D filtering structure.
• To demonstrate that robustBF outperforms standard and counting Bloom Filters in memory consumption, speed, and accuracy.

Proposed method

• The authors modify the Murmur hash function through experimental evaluation to select the best-performing variant for improved bit distribution and reduced collisions.
• They implement a 2D Bloom Filter structure using two prime-numbered dimensions (X and Y) to minimize false positives and ensure even distribution of set bits.
• The 2D array is constructed such that X = P_{i+3} and Y = P_{i-3}, where P_i is a precomputed list of prime numbers, ensuring optimal array dimensions.
• The filter uses exactly half the number of optimal hash functions derived from the theoretical formula, leveraging modulus operations on X and Y for index calculation.
• Memory is allocated as an unsigned long int 2D array with β bits per cell, and the total memory is X × Y × β bits.
• The system uses a hybrid approach: optimized hashing for input mapping and a 2D array structure to reduce false positives while maintaining fast insertion and lookup.

Experimental results

Research questions

• RQ1 Can a 2D Bloom Filter with optimized hashing achieve near-zero false positive probability while significantly reducing memory usage?
• RQ2 How does the choice of hash function affect the false positive rate and performance in a 2D Bloom Filter?
• RQ3 Can a 2D Bloom Filter with prime-numbered dimensions outperform standard and counting Bloom Filters in memory efficiency and accuracy?
• RQ4 To what extent can memory consumption be reduced without compromising lookup or insertion speed?
• RQ5 Is it possible to achieve near-100% accuracy in membership filtering with minimal memory overhead using a novel 2D structure?

Key findings

• robustBF reduces memory consumption by 10.40× on average compared to the standard Bloom Filter (SBF) and 44.01× compared to the counting Bloom Filter (CBF).
• The false positive probability of robustBF is nearly zero (0) across all test sets, including disjoint and random sets, whereas CBF exhibits high false positives (up to 0.995) in such cases.
• robustBF is 2.038× faster than SBF and 2.48× faster than CBF in inserting 10 million items.
• For 10 million items, robustBF uses only 1.55MB of memory, compared to 24MB used by CBF, demonstrating a 15.5× memory reduction.
• The 2D structure with prime-numbered dimensions (X and Y) significantly reduces false positives, especially in worst-case data patterns like disjoint and random sets.
• The filter achieves near-100% accuracy with a target false positive probability of 0.001, making it the first Bloom Filter variant to combine high accuracy with drastically reduced memory footprint.