[Paper Review] Building a serverless Data Lakehouse from spare parts

Jacopo Tagliabue, Ciro Greco+1 more|arXiv (Cornell University)|Aug 10, 2023

Data Quality and ManagementDecision Sciences3 citations

TL;DR

This paper presents Bauplan, a serverless Data Lakehouse built by repurposing open-source components rather than rewriting from scratch or relying on legacy Big Data tools. By combining existing systems like Apache Iceberg, DuckDB, and custom containerized runtimes, the platform delivers a unified, developer-centric experience for both interactive querying and scalable pipeline execution with low-latency function execution and efficient caching.

ABSTRACT

The recently proposed Data Lakehouse architecture is built on open file formats, performance, and first-class support for data transformation, BI and data science: while the vision stresses the importance of lowering the barrier for data work, existing implementations often struggle to live up to user expectations. At Bauplan, we decided to build a new serverless platform to fulfill the Lakehouse vision. Since building from scratch is a challenge unfit for a startup, we started by re-using (sometimes unconventionally) existing projects, and then investing in improving the areas that would give us the highest marginal gains for the developer experience. In this work, we review user experience, high-level architecture and tooling decisions, and conclude by sharing plans for future development.

Motivation & Objective

• To address the gap between the idealized vision of Data Lakehouses and the poor developer experience in existing implementations.
• To enable a flexible, scalable, and low-friction data platform that supports both interactive querying and production-grade data pipelines.
• To reduce development overhead by reusing existing open-source tools instead of building from scratch or retrofitting legacy Big Data frameworks.
• To improve developer ergonomics through a CLI-first design that supports synchronous and asynchronous workflows across development and production.
• To explore a pragmatic path to implementing the Data Lakehouse vision that balances innovation with technical feasibility for startups.

Proposed method

• The platform reuses established open-source projects such as Apache Iceberg for table format, DuckDB for SQL engine, and Nessie for version control, treating them as foundational 'Lego bricks'.
• For Python execution, custom containerized runtimes with a local, disk-based cache are built to optimize startup time and reduce dependency download overhead using power-law package utilization patterns.
• A lightweight container manager enables function-level isolation and customizable sharing policies within DAG executions, supporting both synchronous and asynchronous modes.
• The system leverages object storage for caching and supports serverless execution with fast startup (300ms) and minimal overhead, especially in single-tenant, 'Bring Your Own Cloud' deployments.
• The CLI-first interface provides two core commands: 'bauplan query' for synchronous, point-in-time data exploration and 'bauplan run' for asynchronous, DAG-based pipeline execution with replay and debugging capabilities.
• Time-travel semantics are natively supported via branch-aware queries, allowing users to inspect data at different points in the version history.

Experimental results

Research questions

• RQ1 Can a serverless Data Lakehouse be effectively built by reusing existing open-source components rather than rewriting from scratch or extending legacy Big Data frameworks?
• RQ2 How can a consistent developer experience be achieved across both interactive querying and production pipeline execution in a composable, serverless architecture?
• RQ3 What performance and ergonomics trade-offs arise when using containerized runtimes for Python functions in a serverless data platform?
• RQ4 To what extent can caching strategies based on package utilization patterns reduce cold-start overhead in serverless data functions?
• RQ5 How can versioned data, time-travel, and reproducible pipelines be natively supported in a composable data management system?

Key findings

• The platform achieves a fast startup time of 300ms for serverless Python functions through efficient, local, disk-based caching of commonly used packages.
• By leveraging the power-law distribution of package usage, the caching strategy significantly reduces overall download times and improves function initialization performance.
• The CLI-first design enables seamless integration with existing developer workflows, supporting both interactive exploration and complex, reproducible pipeline execution.
• The system supports both synchronous and asynchronous execution modes across development and production, with consistent semantics and full versioning via Nessie.
• The use of containerized runtimes with fine-grained isolation and customizable sharing policies enables secure, scalable execution of data pipelines within a single DAG.
• The approach of reusing open-source components allowed the team to rapidly converge on a working, end-to-end system and validate core assumptions with early adopters.