[Paper Review] numpywren: serverless linear algebra

Vaishaal Shankar, Karl Krauth+6 more|arXiv (Cornell University)|Oct 23, 2018

Cloud Computing and Resource Management28 references67 citations

TL;DR

numpywren enables large-scale linear algebra with serverless compute, achieving close performance to ScaLAPACK for key algorithms and significantly better compute efficiency, while highlighting serverless locality limits.

ABSTRACT

Linear algebra operations are widely used in scientific computing and machine learning applications. However, it is challenging for scientists and data analysts to run linear algebra at scales beyond a single machine. Traditional approaches either require access to supercomputing clusters, or impose configuration and cluster management challenges. In this paper we show how the disaggregation of storage and compute resources in so-called "serverless" environments, combined with compute-intensive workload characteristics, can be exploited to achieve elastic scalability and ease of management. We present numpywren, a system for linear algebra built on a serverless architecture. We also introduce LAmbdaPACK, a domain-specific language designed to implement highly parallel linear algebra algorithms in a serverless setting. We show that, for certain linear algebra algorithms such as matrix multiply, singular value decomposition, and Cholesky decomposition, numpywren's performance (completion time) is within 33% of ScaLAPACK, and its compute efficiency (total CPU-hours) is up to 240% better due to elasticity, while providing an easier to use interface and better fault tolerance. At the same time, we show that the inability of serverless runtimes to exploit locality across the cores in a machine fundamentally limits their network efficiency, which limits performance on other algorithms such as QR factorization. This highlights how cloud providers could better support these types of computations through small changes in their infrastructure.

Motivation & Objective

• Motivate the need for scalable linear algebra beyond single machines and reduce cluster provisioning complexity.
• Propose a serverless architecture that disaggregates storage and compute for elastic scalability in linear algebra workloads.
• Introduce LAmbdaPACK, a DSL to express parallel linear algebra on tiled matrices in a stateless setting.
• Demonstrate performance and fault-tolerance benefits relative to traditional HPC and fault-tolerant data-parallel systems.

Proposed method

• Develop a serverless system (numpywren) that runs linear algebra tasks as stateless functions and uses a distributed object store for intermediate state.
• Introduce LAmbdaPACK, a domain-specific language to express tiled linear algebra algorithms as DAG-like dependency graphs.
• Use decentralized dependency analysis to generate executable task graphs from LAmbdaPACK programs.
• Implement a fault-tolerant execution model with leases and an elastic provisioner to manage workers.
• Evaluate end-to-end performance on GEMM, QR, SVD, and Cholesky against ScaLAPACK and Dask.
• Discuss limitations due to lack of locality exploitation in serverless runtimes and potential infrastructure tweaks.

Experimental results

Research questions

• RQ1 Can serverless runtimes efficiently execute large-scale linear algebra with disaggregated storage?
• RQ2 How close can serverless linear algebra get to traditional HPC libraries in completion time and CPU-hours?
• RQ3 What are the trade-offs of stateless task design on network traffic and fault tolerance for linear algebra algorithms?
• RQ4 How does LAmbdaPACK enable compact representation and scalable scheduling of complex linear algebra DAGs?

Key findings

• Numpywren’s performance for matrix multiply, SVD, and Cholesky decomposition is within 33% of ScaLAPACK.
• Compute efficiency is up to 240% better due to elasticity.
• For Cholesky on a 1M x 1M matrix, numpywren is within 36% of ScaLAPACK completion time and can use 33% fewer CPU-hours.
• Compared to Dask, numpywren can be up to 320% faster on fault-tolerant data-parallel workloads.
• Serverless locality limitations reduce network efficiency for certain algorithms such as QR factorization, highlighting infrastructure design opportunities for providers.
• LAmbdaPACK enables compact, large-scale DAG representations (millions of nodes in ~2 KB) and supports key algorithms like Cholesky, TSQR, LU, and SVD.