[Paper Review] Occupy the Cloud: Distributed Computing for the 99%

Eric Jonas, Qifan Pu+3 more|arXiv (Cornell University)|Feb 13, 2017

Cloud Computing and Resource Management29 references57 citations

TL;DR

The paper proposes a serverless, stateless-function model (PyWren) for distributed computing to simplify elasticity and deployment, showing it can implement diverse paradigms such as MapReduce and BSP with remote storage as the state. It benchmarks performance and discusses system challenges and future directions.

ABSTRACT

Distributed computing remains inaccessible to a large number of users, in spite of many open source platforms and extensive commercial offerings. While distributed computation frameworks have moved beyond a simple map-reduce model, many users are still left to struggle with complex cluster management and configuration tools, even for running simple embarrassingly parallel jobs. We argue that stateless functions represent a viable platform for these users, eliminating cluster management overhead, fulfilling the promise of elasticity. Furthermore, using our prototype implementation, PyWren, we show that this model is general enough to implement a number of distributed computing models, such as BSP, efficiently. Extrapolating from recent trends in network bandwidth and the advent of disaggregated storage, we suggest that stateless functions are a natural fit for data processing in future computing environments.

Motivation & Objective

• Motivate making cloud computing accessible to a broad user base by removing cluster management overhead.
• Propose a serverless architecture using stateless functions and remote storage as the core abstraction for data processing.
• Demonstrate through PyWren that diverse distributed models (e.g., MapReduce, BSP, parameter servers) can be implemented efficiently with minimal state.
• Assess performance trade-offs of using purely remote storage for inputs/outputs and identify bottlenecks and future challenges.

Proposed method

• Introduce a serverless execution model with stateless functions and a global scheduler.
• Develop PyWren to serialize and deploy user Python functions to AWS Lambda via S3, enabling a map primitive.
• Benchmark IO and compute using remote object storage (S3) and per-core throughput, showing scalability across thousands of functions.
• Demonstrate BSP-style and MapReduce-like workloads and compare with Spark on dedicated clusters.
• Discuss generality by constructing higher-level abstractions (Map+Reduce, BSP, parameter servers) atop stateless functions.

Experimental results

Research questions

• RQ1 Can stateless functions plus remote storage provide sufficient performance for embarrassingly parallel and moderately coupled workloads?
• RQ2 What are the performance characteristics (compute and I/O) when using remote storage as the sole state medium?
• RQ3 To what extent can common distributed computing abstractions (MapReduce, BSP, parameter servers) be implemented on a stateless-function platform?
• RQ4 What are the practical system bottlenecks (launch overhead, storage throughput, scheduling) and how might they be mitigated?
• RQ5 How does the cost and elasticity of a serverless approach compare to traditional clusters for large-scale analytics?

Key findings

• Stateless-function execution with remote storage achieves 30-40 MB/s write/read per core to S3, and scales to 60-80 GB/s across 2800 simultaneous functions.
• A 1TB sort benchmark can be completed with PyWren using distributed Lambda workers and Redis sharding, illustrating scalable shuffle-enabled performance.
• A 83M-item word-count workload is only about 17% slower than PySpark on dedicated servers, demonstrating competitive performance for large-scale tasks.
• Matrix multiplication within each Lambda yields 18 GFLOPS per core, scaling beyond 40 TFLOPS with 2800 workers.
• Shuffle-intensive workloads demonstrate feasibility, but storage throughput becomes a major bottleneck as data movement grows.
• Parameter-server workloads (e.g., Hogwild!) can be implemented with stateless functions using a central key-value store for coordination.