[Paper Review] Matrix Factorization at Scale: a Comparison of Scientific Data Analytics in Spark and C+MPI Using Three Case Studies

Alex Gittens, Aditya Devarakonda+15 more|arXiv (Cornell University)|Jul 5, 2016

Advanced Data Storage Technologies42 references16 citations

TL;DR

This paper evaluates Apache Spark and C+MPI for large-scale matrix factorization in scientific computing, implementing PCA, NMF, and CX on TB-sized datasets from particle physics, climate modeling, and bioimaging. Spark achieves strong scaling up to 1600 nodes but exhibits a 2×–25× performance gap due to scheduling overhead, stragglers, and serialization, highlighting critical bottlenecks for HPC workloads.

ABSTRACT

We explore the trade-offs of performing linear algebra using Apache Spark, compared to traditional C and MPI implementations on HPC platforms. Spark is designed for data analytics on cluster computing platforms with access to local disks and is optimized for data-parallel tasks. We examine three widely-used and important matrix factorizations: NMF (for physical plausability), PCA (for its ubiquity) and CX (for data interpretability). We apply these methods to TB-sized problems in particle physics, climate modeling and bioimaging. The data matrices are tall-and-skinny which enable the algorithms to map conveniently into Spark's data-parallel model. We perform scaling experiments on up to 1600 Cray XC40 nodes, describe the sources of slowdowns, and provide tuning guidance to obtain high performance.

Motivation & Objective

• To evaluate the feasibility and performance of Apache Spark for large-scale scientific matrix factorizations compared to traditional HPC-optimized C+MPI codes.
• To address the performance gap between modern data analytics frameworks like Spark and high-performance computing (HPC) stacks in real-world scientific workloads.
• To identify key performance bottlenecks in Spark—such as straggler effects, task scheduling, and serialization—when applied to linear algebra workloads at scale.
• To provide tuning guidance and architectural insights for improving Spark’s performance on HPC platforms for scientific data analytics.
• To assess whether Spark can effectively scale for TB-sized scientific datasets while maintaining performance comparable to hand-optimized HPC codes.

Proposed method

• Implemented parallel versions of three matrix factorizations—PCA, NMF, and CX—using both Spark and C+MPI on HPC hardware.
• Used TB-sized datasets from three scientific domains: Daya Bay neutrino experiment (1.6TB), CAM5 climate simulation (16TB), and mass spectrometry imaging (2.2TB).
• Employed Spark’s resilient distributed datasets (RDDs) and bulk-synchronous processing (BSP) model to express data-parallel operations on tall-and-skinny matrices.
• Conducted strong scaling experiments on up to 1600 nodes of a Cray XC40 system to measure end-to-end performance and identify scaling bottlenecks.
• Measured and analyzed performance overheads from task scheduling, straggler effects, result serialization, and task deserialization in Spark.
• Used speculation in Spark to mitigate straggler effects, but found it ineffective due to high inter-worker data fetch latency.

Experimental results

Research questions

• RQ1 How does Spark’s performance compare to C+MPI for large-scale matrix factorizations such as PCA, NMF, and CX on TB-sized scientific datasets?
• RQ2 What are the dominant performance bottlenecks in Spark when scaling to 1600 nodes for scientific linear algebra workloads?
• RQ3 To what extent do straggler effects and task scheduling overheads limit Spark’s scalability in data-parallel scientific computing?
• RQ4 Can Spark effectively leverage high-performance linear algebra libraries when interfaced with MPI-based runtimes, and what are the data transfer costs?
• RQ5 How does I/O performance impact end-to-end execution time in large-scale data analytics workloads on HPC systems?

Key findings

• Spark successfully scaled to 1600 nodes for PCA, NMF, and CX on TB-sized datasets, demonstrating feasibility for large-scale scientific analytics.
• The end-to-end performance gap between Spark and C+MPI ranged from 2× to 25×, with the gap widening to 10×–40× when I/O was excluded.
• Straggler effects contributed significantly to overhead, with idle times reaching O(100) seconds across multiple PCA iterations due to synchronization barriers in Spark’s BSP model.
• Task scheduling, result serialization, and deserialization overheads dominated runtime by an order of magnitude at scale, exceeding computational costs.
• Enabling Spark’s speculation feature had no measurable performance improvement due to high latency in fetching RDD partitions from remote workers.
• Variability in runtimes reached up to 25% (median 645s, min 489s, max 716s), with no consistent stage responsible for delays, indicating systemic instability in task execution timing.