[Paper Review] Apiary: A DBMS-Integrated Transactional Function-as-a-Service Framework

Peter Kraft, Qian Li+17 more|arXiv (Cornell University)|Aug 27, 2022

Cloud Computing and Resource Management4 citations

TL;DR

Apiary introduces a DBMS-integrated serverless framework that co-locates function execution and data management within a distributed database engine, eliminating the performance and consistency overhead of traditional FaaS by using the DBMS as a unified runtime for functions, data, and logging. It achieves 2–68x performance improvement over existing FaaS platforms on microservice workloads by reducing communication latency and enabling efficient, fault-tolerant function composition with end-to-end transactional guarantees.

ABSTRACT

Developers increasingly use function-as-a-service (FaaS) platforms for data-centric applications that perform low-latency and transactional operations on data, such as for microservices or web serving. Unfortunately, existing FaaS platforms support these applications poorly because they physically and logically separate application logic, executed in cloud functions, from data management, done in interactive transactions accessing remote storage. Physical separation harms performance while logical separation complicates efficiently providing transactional guarantees and fault tolerance. This paper introduces Apiary, a novel DBMS-integrated FaaS platform for deploying and composing fault-tolerant transactional functions. Apiary physically co-locates and logically integrates function execution and data management by wrapping a distributed DBMS engine and using it as a unified runtime for function execution, data management, and operational logging, thus providing similar or stronger transactional guarantees as comparable systems while greatly improving performance and observability. To allow developers to write complex stateful programs, we leverage this integration to enable efficient and fault-tolerant function composition, building a frontend for orchestrating workflows of functions with the guarantees that each workflow runs to completion and each function in a workflow executes exactly once. We evaluate Apiary against research and production FaaS platforms and show it outperforms them by 2--68x on microservice workloads by reducing communication overhead.

Motivation & Objective

• To address the performance and consistency limitations of existing FaaS platforms that separate application logic from data management.
• To eliminate the communication and coordination overhead caused by physical and logical separation between functions and data stores in traditional FaaS.
• To provide strong transactional guarantees—such as exactly-once execution and atomic workflow composition—within a serverless execution model.
• To enable efficient, fault-tolerant composition of stateful functions while maintaining low-latency and high-throughput execution.
• To unify function execution, data management, and operational logging under a single DBMS runtime to improve observability and system coherence.

Proposed method

• Apiary integrates a distributed DBMS engine as the core runtime, using it to execute functions, manage data, and log operations in a single, consistent system.
• It leverages the DBMS’s ACID properties and internal logging mechanisms to natively support transactional execution of serverless functions with exactly-once semantics.
• Function composition is implemented via a workflow frontend that orchestrates sequences of functions, ensuring each function executes exactly once and the entire workflow is atomic.
• The system co-locates function execution and data access within the same physical nodes as the DBMS, minimizing network I/O and latency.
• It uses the DBMS’s MVCC and logging infrastructure to provide fault tolerance and crash consistency without requiring external coordination.
• The architecture enables efficient stateful computation by persisting function state and intermediate results directly in the DBMS, reducing external storage access.

Experimental results

Research questions

• RQ1 Can a DBMS-native execution model eliminate the performance and consistency bottlenecks of traditional FaaS platforms that decouple functions from data storage?
• RQ2 How can strong transactional guarantees—such as exactly-once execution and atomic workflow composition—be efficiently enforced in a serverless environment?
• RQ3 To what extent can a unified DBMS runtime reduce communication overhead and improve performance in stateful, transactional microservices?
• RQ4 Can the integration of function execution, data management, and logging within a single DBMS engine improve system observability and fault tolerance?
• RQ5 What performance and scalability gains can be achieved by co-locating function execution with data storage in a distributed database system?

Key findings

• Apiary achieves 2–68x better performance than research and production FaaS platforms on microservice workloads by minimizing inter-component communication and leveraging in-memory data access.
• The system reduces end-to-end latency by co-locating function execution with data storage, eliminating network round trips between function and database services.
• Apiary ensures exactly-once execution semantics for all functions in a workflow, with atomic commit guarantees across multiple function invocations.
• The integration with the DBMS enables efficient fault tolerance and crash recovery through native logging and MVCC mechanisms.
• Observability is significantly improved due to unified logging and transaction tracing within the DBMS, simplifying debugging and monitoring.
• The framework supports complex, stateful workflows with minimal performance degradation, demonstrating strong scalability and consistency in production-like workloads.