[Paper Review] Partitioned Learned Bloom Filter

Kapil Vaidya, Eric Knorr+2 more|arXiv (Cornell University)|Jun 5, 2020

Caching and Content Delivery17 references4 citations

TL;DR

This paper proposes the Partitioned Learned Bloom Filter (PLBF), a novel framework that optimizes learned Bloom filters by partitioning the model score space into multiple regions, each with a dedicated backup Bloom filter tuned for minimal space usage. By formulating the problem as an optimization task to minimize space while meeting a target false positive rate, PLBF achieves up to 26x lower false positive rates than baselines like AdaBF and the sandwiching approach, with significant space savings across synthetic, URL, and EMBER datasets.

ABSTRACT

Bloom filters are space-efficient probabilistic data structures that are used to test whether an element is a member of a set, and may return false positives. Recently, variations referred to as learned Bloom filters were developed that can provide improved performance in terms of the rate of false positives, by using a learned model for the represented set. However, previous methods for learned Bloom filters do not take full advantage of the learned model. Here we show how to frame the problem of optimal model utilization as an optimization problem, and using our framework derive algorithms that can achieve near-optimal performance in many cases. Experimental results from both simulated and real-world datasets show significant performance improvements from our optimization approach over both the original learned Bloom filter constructions and previously proposed heuristic improvements.

Motivation & Objective

• To address the limitations of prior learned Bloom filter designs that rely on heuristic threshold selection and single-region partitioning.
• To improve space efficiency by leveraging multiple score partitions, each with a tailored backup Bloom filter.
• To formulate the optimal partitioning and Bloom filter configuration as a constrained optimization problem for minimal space usage under a target false positive rate.
• To demonstrate that PLBF outperforms existing methods—both heuristic and optimal—across diverse real-world and synthetic datasets.
• To provide a general, principled framework for model-driven data structure design that surpasses theoretical lower bounds for standard Bloom filters.

Proposed method

• The method partitions the learned model's score output into k discrete regions, enabling region-specific backup Bloom filters.
• For each region, the false positive rate of the backup Bloom filter is optimized via dynamic programming to minimize total space usage under a global false positive rate constraint.
• The optimal threshold values and Bloom filter parameters are derived by solving a constrained optimization problem that balances model score distribution and space trade-offs.
• The framework uses dynamic programming discretization (DP algorithm) to efficiently compute the optimal partitioning and filter configurations.
• The approach is evaluated using real datasets (URLs, EMBER) and synthetic data with Zipfian score distributions, with model performance measured via F1 score and space/FPR trade-offs.
• The method generalizes prior work by allowing multiple partitions and optimal configuration, avoiding heuristic threshold and filter size choices.

Experimental results

Research questions

• RQ1 Can a learned Bloom filter framework that uses multiple score partitions and region-specific backup Bloom filters achieve better space efficiency than single-threshold or heuristic multi-threshold designs?
• RQ2 How does the number of score partitions (k) affect the space/FPR trade-off, and what is the optimal k for practical performance?
• RQ3 To what extent can the proposed optimization framework reduce false positive rates compared to standard and learned Bloom filter baselines, especially when the model is highly accurate?
• RQ4 Does the framework maintain or improve performance when the model's predictive quality varies, as seen in real-world datasets like EMBER?
• RQ5 Can the optimization framework achieve space savings beyond theoretical lower bounds for standard Bloom filters by exploiting data-specific structure?

Key findings

• PLBF achieves up to 26x lower false positive rates than the sandwiching approach and 9x lower than AdaBF on the URLs dataset, using the same space (500KB).
• For a target false positive rate of 0.001, PLBF uses 6x less space than AdaBF and 8.8x less than the sandwiching approach on the synthetic dataset.
• On the EMBER dataset, where model accuracy is lower (F1 = 0.85), PLBF still achieves 1.9x better FPR than AdaBF and 3x better than the sandwiching approach at the same space.
• Space savings increase with the number of partitions (k), but diminishing returns are observed beyond 4–6 regions, with no significant gain at k=25.
• The method consistently outperforms all baselines across all datasets, with the largest gains on synthetic and URL data where model score distributions are well-separated.
• The KL divergence between key and non-key score distributions is a key determinant of space savings, and PLBF's optimization framework maximizes this gain.