[Paper Review] HRDBMS: Combining the Best of Modern and Traditional Relational Databases

Jason Arnold, Boris Glavic+1 more|arXiv (Cornell University)|Jan 24, 2019

Advanced Database Systems and Queries9 references4 citations

TL;DR

HRDBMS introduces a hybrid distributed relational database that unifies the query optimization and ACID compliance of traditional RDBMS with the scalability of Big Data platforms like Hive and Spark. By using a purpose-built execution engine tailored for SQL workloads, it eliminates performance bottlenecks such as excessive intermediate materialization, lack of statistics, and rigid communication patterns, achieving superior performance on complex analytical queries while maintaining full ACID properties.

ABSTRACT

HRDBMS is a novel distributed relational database that uses a hybrid model combining the best of traditional distributed relational databases and Big Data analytics platforms such as Hive. This allows HRDBMS to leverage years worth of research regarding query optimization, while also taking advantage of the scalability of Big Data platforms. The system uses an execution framework that is tailored for relational processing, thus addressing some of the performance challenges of running SQL on top of platforms such as MapReduce and Spark. These include excessive materialization of intermediate results, lack of a global cost-based optimization, unnecessary sorting, lack of index support, no statistics, no support for DML and ACID, and excessive communication caused by the rigid communication patterns enforced by these platforms.

Motivation & Objective

• To address the performance limitations of running SQL on Big Data platforms like Spark and MapReduce.
• To integrate the robust query optimization and ACID transaction support of traditional RDBMS with the horizontal scalability of modern data platforms.
• To eliminate inefficiencies such as unnecessary sorting, excessive intermediate result materialization, and lack of statistics in existing Big Data SQL engines.
• To design a hybrid execution framework that supports cost-based optimization, indexing, and DML operations in a distributed environment.
• To enable efficient, scalable, and ACID-compliant SQL processing on large-scale data workloads.

Proposed method

• HRDBMS employs a hybrid execution model that combines the strengths of traditional RDBMS query optimization with the distributed scalability of Big Data platforms.
• It uses a purpose-built execution engine optimized for relational processing, avoiding the rigid communication patterns of MapReduce and Spark.
• The system supports global cost-based optimization by maintaining statistics and enabling index usage across distributed nodes.
• It reduces intermediate result materialization by applying pushdown and pruning techniques during query planning.
• HRDBMS supports full DML operations and ACID transactions, ensuring data consistency in distributed environments.
• The architecture enables efficient join processing and avoids unnecessary sorting through adaptive query execution strategies.

Experimental results

Research questions

• RQ1 How can traditional RDBMS query optimization techniques be effectively combined with the scalability of Big Data platforms?
• RQ2 What architectural changes are required to support ACID transactions and statistics in a distributed SQL engine built on Big Data infrastructure?
• RQ3 Can a hybrid execution engine reduce intermediate result materialization and communication overhead compared to MapReduce and Spark?
• RQ4 To what extent can cost-based optimization improve query performance in a distributed SQL system without sacrificing scalability?
• RQ5 How does the integration of indexing and statistics improve performance in a Big Data SQL engine?

Key findings

• HRDBMS achieves significantly lower query execution times compared to Spark SQL and Hive on complex analytical queries by minimizing intermediate result materialization.
• The system demonstrates improved performance through cost-based optimization, enabled by maintained statistics and selective index usage.
• By avoiding rigid communication patterns, HRDBMS reduces network I/O and improves scalability on large-scale data workloads.
• The inclusion of full DML and ACID support enables reliable transaction processing, which is absent in most Big Data SQL engines.
• The hybrid execution framework outperforms existing platforms in both throughput and latency for complex joins and aggregations.
• The system successfully supports pushdown predicates and selective processing, reducing data movement and improving query efficiency.