[Paper Review] A Space-Efficient Dynamic Dictionary for Multisets with Constant Time Operations

Ioana O. Bercea, Guy Even|arXiv (Cornell University)|May 5, 2020

Caching and Content Delivery27 references4 citations

TL;DR

This paper presents the first space-efficient dynamic dictionary for multisets that supports all operations—insertion, deletion, and multiplicity queries—in worst-case constant time with high probability. It achieves this by introducing a novel technique using weighted balls-into-bins experiments with logarithmically weighted balls to efficiently store variable-length binary counters, resolving an open problem posed by Arbitman et al. (2010).

ABSTRACT

We consider the dynamic dictionary problem for multisets. Given an upper bound $n$ on the total cardinality of the multiset (i.e., including multiplicities) at any point in time, the goal is to design a data structure that supports multiplicity queries and allows insertions and deletions to the multiset (i.e., the dynamic setting). The data structure must be space-efficient (the space is $1+o(1)$ times the information-theoretic lower bound) and support all operations in constant time with high probability. In this paper, we present the first dynamic dictionary for multisets that achieves these performance guarantees. This answers an open problem of Arbitman, Naor and Segev (FOCS 2010). The previously best-known construction of Pagh, Pagh and Rao (SODA 2005) supports membership in constant time, multiplicity queries in $O(\log n)$ time in the worst case, and insertions and deletions in constant expected amortized time. The main technical component of our solution is a strategy for efficiently storing variable-length binary counters using weighted balls-into-bins experiments in which balls have logarithmic weights. We also obtain a counting filter that approximates multiplicity queries with a one sided error, using the reduction of Carter et al. (STOC 1978). Counting filters have received significant attention over the years due to their applicability in practice.We present the first counting filter with constant time operations.

Motivation & Objective

• To design a dynamic dictionary for multisets that is space-efficient and supports all operations in constant time with high probability.
• To resolve an open problem posed by Arbitman, Naor, and Segev (2010) on whether a constant-time dynamic multiset dictionary could achieve optimal space usage.
• To extend the state-of-the-art from amortized or expected time to worst-case constant time with high probability, improving upon prior constructions with $O(\log n)$ multiplicity query time.
• To construct a counting filter with constant-time operations and one-sided error, using the multiset dictionary as a core component.
• To achieve space usage within $1+o(1)$ of the information-theoretic lower bound for multisets, $n\log(u/n) + \Theta(n)$ bits.

Proposed method

• Leverages a weighted balls-into-bins model where balls have logarithmic weights to represent variable-length binary counters efficiently.
• Uses $k$-wise $\delta$-dependent permutations to partition the universe into $M$ parts, ensuring balanced load distribution with high probability.
• Employs a pseudo-random partitioning scheme based on $k'$-wise independent hash functions to map elements to parts while maintaining load balance.
• Applies Chernoff bounds for $k'$-wise independent random variables to prove that each part’s multiset cardinality is bounded by $n^{1/10} + n^{3/40}\log^{3/2}n$ with high probability.
• Introduces a spare storage for elements with high multiplicity (≥ $\log^3 n$) to maintain constant-time operations.
• Reduces the counting filter problem to a multiset dictionary via pairwise independent hashing of elements to fingerprints, enabling constant-time approximate multiplicity queries with one-sided error.

Experimental results

Research questions

• RQ1 Can a dynamic multiset dictionary be constructed that supports all operations in worst-case constant time with high probability while being space-efficient?
• RQ2 Is it possible to achieve space usage within $1+o(1)$ of the information-theoretic lower bound for multisets while maintaining constant-time operations?
• RQ3 Can the techniques used for sets in Arbitman et al. (2010) be extended to handle multisets with arbitrary multiplicities?
• RQ4 What is the maximum cardinality of a multiset in each partition under a pseudo-random partitioning scheme, and can it be bounded with high probability?
• RQ5 Can a counting filter be constructed with constant-time operations and one-sided error, using a dynamic multiset dictionary as a building block?

Key findings

• The proposed dynamic multiset dictionary supports all operations—insertion, deletion, and multiplicity queries—in worst-case constant time with high probability.
• The space usage is $1+o(1)$ times the information-theoretic lower bound of $n\log(u/n) + \Theta(n)$ bits, achieving optimal space efficiency.
• The cardinality of each multiset part after partitioning is bounded by $n^{1/10} + n^{3/40}\log^{3/2}n$ with high probability, ensuring load balance.
• The construction uses a novel weighted balls-into-bins model with logarithmically weighted balls to represent variable-length counters, enabling efficient storage.
• A counting filter is constructed with $O(n\log(1/\varepsilon))$ bits of space and constant-time operations, with one-sided error bounded by $\varepsilon$.
• The result resolves the open problem of Arbitman, Naor, and Segev (2010) by providing the first dynamic multiset dictionary with both optimal space and worst-case constant-time operations.