[Paper Review] Beyond Equi-joins: Ranking, Enumeration and Factorization

Nikolaos Tziavelis, Wolfgang Gatterbauer+1 more|arXiv (Cornell University)|Jan 28, 2021

Game Theory and Voting Systems65 references4 citations

TL;DR

This paper presents the first ranked enumeration algorithm with strong theoretical guarantees for theta-joins involving inequalities, using a novel O(n polylog n)-sized factorized representation (TLFG) to enable efficient, incremental retrieval of the top-k results. It achieves O(n polylog n + k log k) time complexity—within a polylogarithmic factor of optimal—for acyclic full joins, even when output size is n^ℓ, and outperforms state-of-the-art DBMSs by orders of magnitude in practice.

ABSTRACT

We study theta-joins in general and join predicates with conjunctions and disjunctions of inequalities in particular, focusing on ranked enumeration where the answers are returned incrementally in an order dictated by a given ranking function. Our approach achieves strong time and space complexity properties: with $n$ denoting the number of tuples in the database, we guarantee for acyclic full join queries with inequality conditions that for every value of $k$, the $k$ top-ranked answers are returned in $\mathcal{O}(n \operatorname{polylog} n + k \log k)$ time. This is within a polylogarithmic factor of $\mathcal{O}(n + k \log k)$, i.e., the best known complexity for equi-joins, and even of $\mathcal{O}(n+k)$, i.e., the time it takes to look at the input and return $k$ answers in any order. Our guarantees extend to join queries with selections and many types of projections (namely those called "free-connex" queries and those that use bag semantics). Remarkably, they hold even when the number of join results is $n^\ell$ for a join of $\ell$ relations. The key ingredient is a novel $\mathcal{O}(n \operatorname{polylog} n)$-size factorized representation of the query output, which is constructed on-the-fly for a given query and database. In addition to providing the first non-trivial theoretical guarantees beyond equi-joins, we show in an experimental study that our ranked-enumeration approach is also memory-efficient and fast in practice, beating the running time of state-of-the-art database systems by orders of magnitude.

Motivation & Objective

• To address the lack of theoretical time and space complexity guarantees for ranked enumeration in theta-joins with inequality conditions.
• To design an algorithm that returns top-k results incrementally with minimal delay, even when intermediate or final output sizes are large (e.g., n^ℓ for ℓ relations).
• To extend factorized database representations beyond equi-joins to support ranked enumeration under complex predicates, including disjunctions and conjunctions of inequalities.
• To achieve performance close to the theoretical lower bound of O(n + k), even in the worst-case data skew or large output scenarios.
• To demonstrate practical efficiency and scalability through extensive experiments on real and synthetic datasets.

Proposed method

• Introduces a novel factorized representation called the Tuple-List Factor Graph (TLFG) that compactly encodes query results using shared ranges and partitioning strategies.
• Employs a binary or multiway partitioning scheme to recursively decompose the join space, enabling efficient enumeration and ranking.
• Uses a shared ranges abstraction to exploit transitivity in inequality predicates, reducing redundancy and improving factorization efficiency.
• Applies a priority queue-based enumeration strategy that retrieves top-k results in ranking order, with each result generated in O(log k) time after preprocessing.
• Constructs the TLFG representation on-the-fly for a given query and database, ensuring space efficiency proportional to O(n polylog n).
• Supports selections and various projections (e.g., free-connex and bag semantics) by adapting the factorization and enumeration logic accordingly.

Experimental results

Research questions

• RQ1 Can we achieve near-optimal time complexity O(n polylog n + k log k) for ranked enumeration of acyclic full theta-joins with inequality predicates?
• RQ2 Can we design a factorized representation that supports efficient ranked enumeration for complex theta-joins while maintaining space complexity within polylogarithmic factors of O(n)?
• RQ3 How does the proposed method scale with increasing query size, join selectivity, and band-join ranges in real-world workloads?
• RQ4 Can the method outperform traditional DBMSs that materialize the full join result before applying top-k filtering?
• RQ5 What are the theoretical and practical limits of extending factorized representations to general theta-joins beyond equi-joins?

Key findings

• The proposed method achieves O(n polylog n + k log k) time complexity for acyclic full theta-joins with inequality conditions, which is within a polylogarithmic factor of the optimal O(n + k) lower bound.
• The TLFG factorized representation is constructed on-the-fly and uses O(n polylog n) space, enabling efficient access even when the full output size is n^ℓ for ℓ relations.
• The method outperforms state-of-the-art commercial and open-source DBMSs (e.g., PostgreSQL) by orders of magnitude in runtime, especially for small k.
• Experiments on RedditTitles and OceaniaBirds datasets show robust performance across varying query sizes, band-join ranges, and domain sizes, with low delay between consecutive results.
• The approach supports a wide range of queries, including those with selections and free-connex projections, and maintains strong theoretical guarantees even under data skew.
• The TLFG abstraction generalizes and outperforms traditional range tree and partitioning methods, particularly in handling disjunctions and complex inequality predicates.