[Paper Review] FDB: A Query Engine for Factorised Relational Databases

Nurzhan Bakibayev, Dan Olteanu+1 more|arXiv (Cornell University)|Mar 12, 2012

Advanced Database Systems and Queries17 references4 citations

TL;DR

FDB is an in-memory query engine that accelerates select-project-join queries on factorised relational databases using compact, algebraically factorised representations. By exploiting data factorisation through novel optimisation and evaluation algorithms, FDB achieves up to four orders of magnitude performance improvement over traditional relational engines on datasets with many-to-many relationships.

ABSTRACT

Factorised databases are relational databases that use compact factorised representations at the physical layer to reduce data redundancy and boost query performance. This paper introduces FDB, an in-memory query engine for select-project-join queries on factorised databases. Key components of FDB are novel algorithms for query optimisation and evaluation that exploit the succinctness brought by data factorisation. Experiments show that for data sets with many-to-many relationships FDB can outperform relational engines by orders of magnitude.

Motivation & Objective

• To address the performance bottleneck in traditional relational databases caused by data redundancy in many-to-many relationships.
• To enable efficient query processing on factorised relational data by designing a novel in-memory query engine.
• To develop query optimisation and evaluation algorithms that exploit the compactness of factorised representations.
• To demonstrate that factorised representations preserve compactness and performance across multiple query operations.
• To compare the performance of FDB against traditional relational databases and optimisers using both flat and factorised data.

Proposed method

• FDB uses a factorisation tree (f-tree) data structure to represent relations as unions of products of singleton attribute values, enabling compact storage and efficient query processing.
• The engine employs a two-phase approach: query optimisation via f-plans that determine the best factorisation order, and evaluation using recursive traversal of the f-tree.
• FDB supports both full-search and greedy heuristic-based optimisers to explore the exponential space of possible f-tree structures, balancing optimality and efficiency.
• The query evaluation engine performs operations like selection and join directly on the f-tree by traversing and rewriting the nested structure without materialising intermediate results.
• The system uses distributivity of product over union and commutativity to reorganise f-trees into more compact forms that reduce evaluation cost.
• FDB evaluates queries on both flat and factorised input data, comparing performance against SQLite, PostgreSQL, and standard RDBMS engines.

Experimental results

Research questions

• RQ1 Can factorised representations significantly reduce query evaluation time and result size compared to traditional relational engines?
• RQ2 How does the quality of f-plans (i.e., factorisation order) affect query performance in FDB?
• RQ3 Does the compactness of factorised representations degrade over multiple query operations?
• RQ4 What is the trade-off between optimality and efficiency in f-tree search space exploration?
• RQ5 How does FDB’s performance scale with increasing query size and data skew?

Key findings

• FDB achieves up to four orders of magnitude faster query evaluation than traditional relational databases on datasets with many-to-many relationships.
• For queries with up to four equality selections, FDB reduces result sizes to under 4,000 singletons even when processing up to 500 million data values.
• The full-search optimiser finds optimal f-plans but scales exponentially with query size, while the greedy heuristic is 2–3 orders of magnitude faster with only up to 50% performance degradation.
• On factorised input data, FDB maintains compact results and achieves significant speedups even when the input size is reduced to 1,000 tuples, where both FDB and RDB run in under 0.1 seconds.
• The performance gap between FDB and RDB closes only when input data size drops below 1,000 tuples, indicating FDB’s advantage is most pronounced on large, highly factorisable data.
• Experiments confirm that factorisation quality does not decay across operations, demonstrating the sustainability of compact representations through multiple query steps.