[Paper Review] Archipelago: A Scalable Low-Latency Serverless Platform

Arjun Singhvi, Kevin Houck+4 more|arXiv (Cornell University)|Nov 22, 2019

Cloud Computing and Resource Management30 references19 citations

TL;DR

Archipelago is a scalable, low-latency serverless platform that partitions a cluster into worker pools managed by semi-global schedulers (SGS), uses latency-aware scheduling with proactive sandbox allocation, and employs sandbox-aware load balancing to route requests. It achieves sub-300ms tail latency for 99% of requests, reducing tail latency by up to 36X compared to state-of-the-art platforms.

ABSTRACT

The increased use of micro-services to build web applications has spurred the rapid growth of Function-as-a-Service (FaaS) or serverless computing platforms. While FaaS simplifies provisioning and scaling for application developers, it introduces new challenges in resource management that need to be handled by the cloud provider. Our analysis of popular serverless workloads indicates that schedulers need to handle functions that are very short-lived, have unpredictable arrival patterns, and require expensive setup of sandboxes. The challenge of running a large number of such functions in a multi-tenant cluster makes existing scheduling frameworks unsuitable. We present Archipelago, a platform that enables low latency request execution in a multi-tenant serverless setting. Archipelago views each application as a DAG of functions, and every DAG in associated with a latency deadline. Archipelago achieves its per-DAG request latency goals by: (1) partitioning a given cluster into a number of smaller worker pools, and associating each pool with a semi-global scheduler (SGS), (2) using a latency-aware scheduler within each SGS along with proactive sandbox allocation to reduce overheads, and (3) using a load balancing layer to route requests for different DAGs to the appropriate SGS, and automatically scale the number of SGSs per DAG. Our testbed results show that Archipelago meets the latency deadline for more than 99% of realistic application request workloads, and reduces tail latencies by up to 36X compared to state-of-the-art serverless platforms.

Motivation & Objective

• To address the challenge of low-latency, high-throughput execution of short-lived, unpredictable, and sandbox-intensive serverless functions in multi-tenant clusters.
• To ensure end-to-end request deadlines are met for DAG-based serverless applications under dynamic workloads.
• To reduce scheduling overhead and cold starts in large-scale serverless environments through scalable, decentralized scheduling.
• To enable automatic, sandbox-aware scaling of schedulers based on workload demand and latency requirements.

Proposed method

• Archipelago partitions a cluster into multiple worker pools, each managed by a semi-global scheduler (SGS), reducing scheduling overhead and improving scalability.
• Each SGS uses a latency-aware scheduling algorithm that computes 'slack' (remaining time) for each DAG request and applies a shortest-remaining-time-first (SRTF) variant to prioritize high-priority, low-slack requests.
• Proactive sandbox allocation is employed within each SGS, pre-allocating soft-state sandboxes based on DAG request rates to minimize cold starts and setup overhead.
• A sandbox-aware load balancer routes incoming requests to the appropriate SGS based on available sandboxes and workload distribution, enabling dynamic scaling of SGS instances per DAG.
• The system uses soft-state sandboxes that consume only memory and can be evicted without correctness loss, enabling efficient resource reuse.
• The platform supports multi-tenancy by allowing diverse applications with different latency requirements to coexist, with strict deadline guarantees for each DAG.

Experimental results

Research questions

• RQ1 How can a serverless platform achieve low tail latency for short-lived, highly dynamic workloads in a multi-tenant cluster?
• RQ2 What is the optimal trade-off between scheduler granularity and scheduling overhead in large-scale serverless systems?
• RQ3 Can proactive sandbox allocation significantly reduce cold starts and improve end-to-end latency in serverless execution?
• RQ4 How can a decentralized scheduling architecture maintain low latency while ensuring deadline compliance for DAG-based workloads?
• RQ5 What role does sandbox-aware load balancing play in enabling automatic scaling and low-latency request routing?

Key findings

• Archipelago meets the end-to-end latency deadline for more than 99% of realistic application request workloads under dynamic, bursty traffic patterns.
• The platform reduces tail latency by up to 36X compared to state-of-the-art serverless platforms, significantly improving responsiveness.
• Fine-grained partitioning (e.g., 20 SGSs with 1 worker each) increases tail latency by ~4X due to excessive cold starts and scaling overhead.
• Beyond 64 machines per SGS, scheduling overhead begins to dominate, leading to unnecessary scaling and underutilization of workers.
• Proactive sandbox allocation reduces the number of cold starts and improves request completion times, especially under high request variability.
• The sandbox-aware load balancer enables efficient, automatic scaling of SGS instances per DAG, adapting to request arrival rates and sandbox availability.