[Paper Review] Apache Calcite: A Foundational Framework for Optimized Query Processing Over Heterogeneous Data Sources

Edmon Begoli, Jesús Camacho-Rodríguez+3 more|OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information)|Feb 28, 2018

Data Management and Algorithms29 references49 citations

TL;DR

This paper presents Apache Calcite as a modular, embeddable framework that provides query processing, optimization, and language support to federate and optimize queries across heterogeneous data sources and backends.

ABSTRACT

Apache Calcite is a foundational software framework that provides query processing, optimization, and query language support to many popular open-source data processing systems such as Apache Hive, Apache Storm, Apache Flink, Druid, and MapD. Calcite's architecture consists of a modular and extensible query optimizer with hundreds of built-in optimization rules, a query processor capable of processing a variety of query languages, an adapter architecture designed for extensibility, and support for heterogeneous data models and stores (relational, semi-structured, streaming, and geospatial). This flexible, embeddable, and extensible architecture is what makes Calcite an attractive choice for adoption in big-data frameworks. It is an active project that continues to introduce support for the new types of data sources, query languages, and approaches to query processing and optimization.

Motivation & Objective

• Motivate the need for a unifying framework to optimize queries across heterogeneous data sources and backends.
• Describe Calcite’s architecture as a modular, extensible foundation for query processing without storage management.
• Explain how adapters, relational algebra, and traits enable cross-system optimization and execution.
• Highlight the extensibility for multiple data models (relational, semi-structured, streaming, geospatial) and query languages.
• Show adoption and impact through integration with popular open-source systems.

Proposed method

• Describe Calcite’s architecture with a relational algebra core and a pluggable optimizer (rules, metadata providers, planner engines).
• Explain the adapter design (model, schema, schema factory) for reading external data stores and pushing down logic to backends.
• Detail the trait-based physical properties and calling conventions that enable cross-backend optimization.
• Present planner engines (cost-based and exhaustive) and materialized-view rewriting for optimization.
• Discuss extensions for semi-structured, streaming, and geospatial data, plus language-integrated querying (LINQ4J).
• Illustrate integration with industry systems and the role of Calcite as an embeddable optimization layer.

Experimental results

Research questions

• RQ1 How can a single framework support efficient query optimization across heterogeneous data sources and processing backends?
• RQ2 What architectural components are required to enable extensible adapters and cross-system planning?
• RQ3 How can SQL and non-SQL query processing be unified within a common relational algebra core?
• RQ4 What extensions are needed to support semi-structured, streaming, and geospatial data?
• RQ5 What is the impact and adoption of Calcite in industry and academia?

Key findings

• Calcite provides a modular, extensible optimizer with hundreds of rules and pluggable engines.
• Adapters define how Calcite reads diverse data sources and enable cross-backend pushdown of predicates and operations.
• It supports ANSI SQL and extensions, along with a JDBC driver and a query algebra for heterogeneous data models.
• The framework uses traits and calling conventions to reason about physical properties and execution strategies across backends.
• Calcite is widely adopted by projects like Hive, Drill, Storm, Flink, Solr, and many others, illustrating broad industry impact.
• Calcite extends beyond SQL to semi-structured, streaming, and geospatial data and offers language-integrated querying via LINQ4J.