[Paper Review] Join Size Bounds using Lp-Norms on Degree Sequences

Mahmoud Abo Khamis, Vasileios Nakos+2 more|arXiv (Cornell University)|Jun 24, 2023

Data Management and Algorithms4 citations

TL;DR

This paper introduces novel upper bounds on join query output sizes using ℓₚ-norms of degree sequences, significantly tightening existing bounds—especially for acyclic queries—by leveraging information theory and linear programming. The method generalizes prior AGM and PANDA bounds, offering tighter, provably optimal estimates using precomputed ℓₚ-norms of join attributes, with a matching query evaluation algorithm and exponential-time computability.

ABSTRACT

Estimating the output size of a query is a fundamental yet longstanding problem in database query processing. Traditional cardinality estimators used by database systems can routinely underestimate the true output size by orders of magnitude, which leads to significant system performance penalty. Recently, upper bounds have been proposed that are based on information inequalities and incorporate sizes and max-degrees from input relations, yet they their main benefit is limited to cyclic queries, because they degenerate to rather trivial formulas on acyclic queries. We introduce a significant extension of the upper bounds, by incorporating $\ell_p$-norms of the degree sequences of join attributes. Our bounds are significantly lower than previously known bounds, even when applied to acyclic queries. These bounds are also based on information theory, they come with a matching query evaluation algorithm, are computable in exponential time in the query size, and are provably tight when all degrees are "simple".

Motivation & Objective

• Address the persistent problem of cardinality estimation in database query optimization, where traditional estimators often severely underestimate output sizes.
• Overcome the limitation of existing theoretical bounds like AGM and PANDA, which degenerate to trivial formulas on acyclic queries.
• Develop tighter, non-trivial upper bounds on query output size that apply uniformly to both cyclic and acyclic queries.
• Introduce a new class of statistics—ℓₚ-norms of degree sequences—that generalize cardinality and max-degree information for improved estimation accuracy.
• Provide a provably tight, information-theoretic bound with a corresponding query evaluation algorithm computable in exponential time in query size.

Proposed method

• Formulate upper bounds on join output size using ℓₚ-norms of degree sequences of join attributes, where the degree sequence is the sorted list of attribute frequencies.
• Use information inequalities and linear programming to derive the optimal upper bound, with the solution being the minimal value satisfying entropy constraints.
• Generalize previous bounds: ℓ₁-norm corresponds to relation size, ℓ∞-norm to max degree, and intermediate ℓₚ-norms (e.g., ℓ₂, ℓ₃) provide tighter bounds.
• Define log-statistics bᵢ as scaled logarithms of ℓₚ-norms of degree sequences, enabling transformation into entropy-like constraints.
• Construct a lattice of closed sets to represent entropy values of attribute sets, and derive log-statistics from polymatroid entropy functions.
• Apply the bound via a linear program over the polymatroid domain, ensuring tightness when all degrees are 'simple'.

Experimental results

Research questions

• RQ1 Can ℓₚ-norms of degree sequences be used to derive tighter upper bounds on join query output size than existing methods?
• RQ2 How do ℓₚ-norm-based bounds compare to AGM and PANDA bounds in terms of tightness, especially on acyclic queries?
• RQ3 Is the proposed bound provably optimal and computable in a practical setting with precomputed statistics?
• RQ4 Can the use of intermediate ℓₚ-norms (e.g., ℓ₂, ℓ₃) yield significantly better cardinality estimates than relying only on ℓ₁ and ℓ∞?
• RQ5 What is the relationship between the proposed bound and the entropy of attribute sets, and how does this enable tighter estimation?

Key findings

• The proposed ℓₚ-norm-based upper bounds are strictly tighter than previous bounds, even on acyclic queries, where AGM and PANDA bounds degenerate to trivial formulas.
• For the triangle query, using ℓ₂- and ℓ₃-norms yields new upper bounds such as |Q| ≤ (||deg_R(Y|X)||₂² · ||deg_S(Z|Y)||₂² · ||deg_T(X|Z)||₂²)^{1/3}, which can be significantly smaller than the PANDA bound.
• The bound is provably tight when all degrees are 'simple', meaning the degree sequence satisfies a specific regularity condition.
• The method generalizes AGM and PANDA bounds: ℓ₁-norm gives the AGM bound, ℓ∞-norm gives the PANDA bound, and intermediate ℓₚ-norms provide a continuous family of tighter bounds.
• Empirical evaluation shows that the minimal bound across multiple ℓₚ-norms (e.g., ℓ₁, ℓ₂, ℓ₃, ℓ∞) yields the tightest estimate, depending on data distribution.
• The bound is not tight for all polymatroids, but the ratio of the actual log-size to the bound is at most 35/36, demonstrating strong asymptotic tightness.