[Paper Review] InferSpark: Statistical Inference at Scale

Zhuoyue Zhao, Jialing Pei+3 more|arXiv (Cornell University)|Jul 7, 2017

Advanced Database Systems and Queries18 references3 citations

TL;DR

InferSpark is a probabilistic programming framework built on Apache Spark that enables scalable statistical inference for custom Bayesian models. It extends Scala with domain-specific constructs to succinctly define models, compiles them into distributed inference code, and leverages Spark’s in-memory processing to accelerate large-scale inference—demonstrating a 70x reduction in code size for LDA compared to hand-coded implementations.

ABSTRACT

The Apache Spark stack has enabled fast large-scale data processing. Despite a rich library of statistical models and inference algorithms, it does not give domain users the ability to develop their own models. The emergence of probabilistic programming languages has showed the promise of developing sophisticated probabilistic models in a succinct and programmatic way. These frameworks have the potential of automatically generating inference algorithms for the user defined models and answering various statistical queries about the model. It is a perfect time to unite these two great directions to produce a programmable big data analysis framework. We thus propose, InferSpark, a probabilistic programming framework on top of Apache Spark. Efficient statistical inference can be easily implemented on this framework and inference process can leverage the distributed main memory processing power of Spark. This framework makes statistical inference on big data possible and speed up the penetration of probabilistic programming into the data engineering domain.

Motivation & Objective

• Address the lack of support for custom statistical model development in scalable big data frameworks like Apache Spark.
• Overcome the limitations of existing probabilistic programming frameworks (e.g., Infer.NET, Church, Figaro) that do not scale to large datasets or distributed environments.
• Enable data scientists and statisticians to define complex Bayesian models in a concise, programmatic way without manually implementing low-level distributed inference algorithms.
• Unify the expressiveness of probabilistic programming with the distributed computing power of Spark to democratize scalable statistical inference.

Proposed method

• Extend Scala with probabilistic programming constructs to allow intuitive model definition using domain-specific syntax for distributions and conditional dependencies.
• Compile user-defined models into optimized Spark-compatible Scala classes and objects that encapsulate inference algorithms.
• Implement a compiler and runtime system that translates high-level model definitions into distributed inference pipelines using Spark’s RDD and Dataset APIs.
• Leverage variational message passing (VMP) for efficient inference in exponential-family conjugate models, such as LDA with Dirichlet priors.
• Expose a clean API for users to input observed data and query posterior distributions or expectations from the model.
• Design the framework to support future extensibility, including support for non-conjugate priors, other inference algorithms (e.g., Gibbs sampling), and plugin-based inference implementation.

Experimental results

Research questions

• RQ1 Can probabilistic programming be effectively integrated into a distributed big data platform like Apache Spark to enable scalable statistical inference?
• RQ2 To what extent can model definition be simplified using domain-specific language extensions in a general-purpose language like Scala?
• RQ3 How does the performance and code complexity of InferSpark compare to hand-optimized implementations in existing libraries like MLlib for standard models such as LDA?
• RQ4 Can the framework support efficient distributed inference for complex Bayesian networks beyond exponential-family conjugate models?

Key findings

• InferSpark enables the definition of the Latent Dirichlet Allocation (LDA) model in just 7 lines of Scala code, compared to 503 lines in MLlib, demonstrating a significant reduction in implementation complexity.
• The framework successfully scales statistical inference to large datasets by leveraging Spark’s distributed in-memory computation, overcoming the memory limitations of single-machine probabilistic frameworks like Infer.NET.
• Empirical evaluation confirms that InferSpark can perform distributed inference efficiently, with inference time and memory usage proportional to data size and model complexity.
• The prototype supports variational message passing (VMP) for exponential-family conjugate models, enabling fast and scalable posterior estimation.
• The system is designed for extensibility, with plans to support non-conjugate priors, Markov networks, and alternative inference algorithms like Gibbs sampling.
• InferSpark opens the door for statisticians and data scientists to build and experiment with custom Bayesian models at scale without low-level distributed systems programming.