[Paper Review] An Optimal Bloom Filter Replacement Based on Matrix Solving

Ely Porat|ArXiv.org|Apr 11, 2008

Caching and Content Delivery7 references8 citations

TL;DR

This paper proposes a novel, space-optimized dictionary data structure that replaces Bloom Filters by using matrix solving over GF(2^k), achieving optimal space usage of nk + o(n) bits for n keys with k-bit values. It enables O(1) query time with only pairwise independent hash functions, offering a significant space reduction over prior methods while maintaining one-sided error (false positives only).

ABSTRACT

We suggest a method for holding a dictionary data structure, which maps keys to values, in the spirit of Bloom Filters. The space requirements of the dictionary we suggest are much smaller than those of a hashtable. We allow storing n keys, each mapped to value which is a string of k bits. Our suggested method requires nk + o(n) bits space to store the dictionary, and O(n) time to produce the data structure, and allows answering a membership query in O(1) memory probes. The dictionary size does not depend on the size of the keys. However, reducing the space requirements of the data structure comes at a certain cost. Our dictionary has a small probability of a one sided error. When attempting to obtain the value for a key that is stored in the dictionary we always get the correct answer. However, when testing for membership of an element that is not stored in the dictionary, we may get an incorrect answer, and when requesting the value of such an element we may get a certain random value. Our method is based on solving equations in GF(2^k) and using several hash functions. Another significant advantage of our suggested method is that we do not require using sophisticated hash functions. We only require pairwise independent hash functions. We also suggest a data structure that requires only nk bits space, has O(n2) preprocessing time, and has a O(log n) query time. However, this data structures requires a uniform hash functions. In order replace a Bloom Filter of n elements with an error proability of 2^{-k}, we require nk + o(n) memory bits, O(1) query time, O(n) preprocessing time, and only pairwise independent hash function. Even the most advanced previously known Bloom Filter would require nk+O(n) space, and a uniform hash functions, so our method is significantly less space consuming especially when k is small.

Motivation & Objective

• To design a dictionary data structure that replaces Bloom Filters with significantly reduced space usage while maintaining fast query performance.
• To minimize space requirements to nk + o(n) bits, approaching the information-theoretic lower bound, for storing n keys mapped to k-bit values.
• To enable O(1) query time using only pairwise independent hash functions, avoiding the need for computationally expensive uniform hash functions required by prior methods.
• To provide a practical alternative to Bloom Filters for applications like password filtering, caching, and distributed storage, where space efficiency is critical.
• To explore a variant with even lower space usage (nk bits) at the cost of O(log n) query time and O(n²) preprocessing, requiring uniform hash functions.

Proposed method

• The method encodes key-value mappings as a system of linear equations over the finite field GF(2^k), solving for values using matrix inversion.
• It uses multiple pairwise independent hash functions to map keys to positions in a matrix, enabling efficient encoding and decoding of values.
• For the O(1) query variant, the system is partitioned into sub-buckets, and inverse matrices are precomputed and stored in a hash table for fast lookup.
• A speed-up is achieved by grouping sub-buckets with identical inverse matrices and reusing matrix-vector multiplications via word-level parallelism.
• For the space-optimized variant, the system solves n sparse equations in n variables using the block Wiedemann algorithm, reducing space to nk bits.
• The construction ensures that valid keys always return correct values, while non-members may return false positives with probability 2^(-k).

Experimental results

Research questions

• RQ1 Can a Bloom Filter replacement be constructed with space usage closer to the information-theoretic minimum, specifically nk + o(n) bits?
• RQ2 Is it possible to achieve O(1) query time with only pairwise independent hash functions, avoiding the need for uniform or universal hash functions?
• RQ3 Can a data structure be designed with nk bits space usage while maintaining acceptable query performance and correctness guarantees?
• RQ4 How can preprocessing time be minimized while maintaining low space and fast query times in a static, error-tolerant dictionary?
• RQ5 Can the construction be made distributable and scalable for large-scale applications like distributed storage or internet caching?

Key findings

• The proposed data structure uses nk + o(n) bits of space, which is optimal up to lower-order terms, significantly reducing space compared to traditional Bloom Filters.
• It achieves O(1) query time with only pairwise independent hash functions, eliminating the need for expensive uniform hash functions used in prior methods.
• The method supports one-sided error: valid keys always return correct values, while non-members return false positives with probability 2^(-k).
• A variant using nk bits space is proposed, but it requires O(log n) query time and O(n²) preprocessing time, relying on uniform hash functions.
• The preprocessing time can be reduced to O(n) using optimized matrix multiplication and word-level parallelism, even when solving many small matrix systems.
• The construction is robust and practical for real-world applications such as password filtering, differential databases, and internet caching, where space and speed are critical.