[Paper Review] Information Theoretic Limits of Cardinality Estimation: Fisher Meets Shannon

Seth Pettie, Dingyu Wang|arXiv (Cornell University)|Jul 16, 2020

Algorithms and Data Compression39 references4 citations

TL;DR

This paper introduces the Fisher-Shannon (Fish) number, a new information-theoretic metric that quantifies the trade-off between sketch space complexity and estimation error in cardinality estimation. It proves that the optimal Fish-number is approximately 1.98016, achieved asymptotically by the Fishmonger sketch and matched by all base-$q$ variants of PCSA, while showing that no linearizable, mergeable sketch can surpass this bound.

ABSTRACT

Estimating the cardinality (number of distinct elements) of a large multiset is a classic problem in streaming and sketching. In this paper we study the intrinsic tradeoff between the space complexity of the sketch and its estimation error. We define a new measure of efficiency for data sketches called the Fisher-Shannon (FiSh) number $\mathcal{H}/\mathcal{I}$. It captures the tension between the limiting Shannon entropy ($\mathcal{H}$) of the sketch and its normalized Fisher information ($\mathcal{I}$) that characterizes the variance of a statistically efficient, asymptotically unbiased estimator. Our aim in introducing the FiSh-number is to build the mathematical machinery necessary to argue for precise optimality, rather than asymptotic optimality, up to large constant factors. Our results are as follows. [1] We prove that all base-$q$ variants of Flajolet and Martin's PCSA sketch have FiSh-number $H_0/I_0 \approx 1.98016$ and that every base-$q$ variant of HyperLogLog has FiSh-number worse than $H_0/I_0$, but that they tend to $H_0/I_0$ in the limit as $q ightarrow \infty$. Here $H_0,I_0$ are precisely defined constants. [2] We describe a sketch called Fishmonger that is based on a smoothed, entropy-compressed variant of PCSA with a different estimator function. Fishmonger processes a multiset of $[U]$ such that at all times, w.h.p., its space is $(1+o(1))(H_0/I_0)m \approx 1.98m$ bits and its standard error is $1/\sqrt{m}$. For example, to achieve a 1% standard error, one needs a little more than 19,800 bits, or $\approx 2.42$ kilobytes. [3] Finally, we give circumstantial evidence that $H_0/I_0$ is the optimum FiSh-number of mergeable sketches for Cardinality Estimation. We define a natural subset of mergeable sketches called linearizable sketches and prove that no member of this class can beat $H_0/I_0$. The popular mergeable sketches are, in fact, also linearizable.

Motivation & Objective

• To establish a fundamental information-theoretic limit for cardinality estimation in the random oracle model.
• To define and analyze a new metric, the Fisher-Shannon (Fish) number, that captures the trade-off between sketch entropy and Fisher information.
• To prove that the Fish-number $H_0/I_0 \approx 1.98016$ is the information-theoretic lower bound for all linearizable, mergeable sketches.
• To design a new sketch, Fishmonger, that achieves this bound up to a $(1+o(1))$ factor in space and standard error.

Proposed method

• Introduces the Fisher-Shannon (Fish) number $\mathcal{H}/\mathcal{I}$, combining Shannon entropy ($\mathcal{H}$) and normalized Fisher information ($\mathcal{I}$) to measure sketch efficiency.
• Analyzes base-$q$ variants of Flajolet-Martin's PCSA and (Hyper) LogLog sketches, showing their Fish-number is worse than $H_0/I_0$ but converges to it as $q \to \infty$.
• Proposes the Fishmonger sketch, based on entropy-compressed, smoothed PCSA with a modified estimator function to achieve near-optimal performance.
• Applies a data processing inequality argument under the linearizability assumption to derive a lower bound on the Fish-number for mergeable sketches.
• Uses asymptotic analysis and concentration inequalities to show that Fishmonger achieves space complexity $O(\log^2 \log U) + (1+o(1))(H_0/I_0)b$ bits with standard error $1/\sqrt{b}$.
• Proves that the Fish-number $H_0/I_0 \approx 1.98016$ is optimal for all linearizable sketches by showing that any such sketch must satisfy $\mathsf{Fish}(A_q) \geq H_0/I_0$.

Experimental results

Research questions

• RQ1 What is the fundamental information-theoretic limit of space-accuracy trade-offs in cardinality estimation sketches?
• RQ2 Can the Fisher-Shannon number serve as a universal metric for evaluating the efficiency of cardinality estimation sketches?
• RQ3 Is the Fish-number $H_0/I_0 \approx 1.98016$ the optimal achievable bound for all linearizable, mergeable sketches?
• RQ4 Can a practical sketch be constructed that asymptotically achieves this optimal Fish-number with low space and standard error?

Key findings

• All base-$q$ variants of Flajolet-Martin's PCSA sketch achieve a Fish-number of exactly $H_0/I_0 \approx 1.98016$, making them information-theoretically optimal.
• All base-$q$ variants of (Hyper) LogLog have a Fish-number strictly worse than $H_0/I_0$, but converge to it as $q \to \infty$.
• The Fishmonger sketch achieves space complexity $O(\log^2 \log U) + (1+o(1))(H_0/I_0)b$ bits and standard error $1/\sqrt{b}$ with high probability.
• The constant $H_0/I_0 \approx 1.98016$ is proven to be the information-theoretic lower bound for all linearizable sketches, meaning no such sketch can achieve better efficiency.
• The Fish-number $H_0/I_0$ is conjectured to be optimal even beyond linearizable sketches, suggesting a universal limit for mergeable cardinality estimators.