[Paper Review] Spinning Fast Iterative Data Flows

Stephan Ewen, Kostas Tzoumas+2 more|arXiv (Cornell University)|Aug 1, 2012

Parallel Computing and Optimization Techniques22 references4 citations

TL;DR

This paper proposes incremental iterations as an extension to parallel dataflow systems, enabling efficient execution of iterative algorithms with sparse computational dependencies—such as graph algorithms—by exploiting mutable state and workset-based updates. Evaluation shows up to two orders of magnitude speedup compared to bulk iteration, making dataflow systems competitive with specialized frameworks while preserving a unified abstraction.

ABSTRACT

Parallel dataflow systems are a central part of most analytic pipelines for big data. The iterative nature of many analysis and machine learning algorithms, however, is still a challenge for current systems. While certain types of bulk iterative algorithms are supported by novel dataflow frameworks, these systems cannot exploit computational dependencies present in many algorithms, such as graph algorithms. As a result, these algorithms are inefficiently executed and have led to specialized systems based on other paradigms, such as message passing or shared memory. We propose a method to integrate incremental iterations, a form of workset iterations, with parallel dataflows. After showing how to integrate bulk iterations into a dataflow system and its optimizer, we present an extension to the programming model for incremental iterations. The extension alleviates for the lack of mutable state in dataflows and allows for exploiting the sparse computational dependencies inherent in many iterative algorithms. The evaluation of a prototypical implementation shows that those aspects lead to up to two orders of magnitude speedup in algorithm runtime, when exploited. In our experiments, the improved dataflow system is highly competitive with specialized systems while maintaining a transparent and unified dataflow abstraction.

Motivation & Objective

• To address the inefficiency of existing dataflow systems in executing iterative algorithms with sparse computational dependencies.
• To enable dataflow systems to efficiently support algorithms like graph processing and machine learning that evolve state incrementally.
• To unify general-purpose dataflow programming with the performance of specialized iterative systems like Pregel and GraphLab.
• To extend dataflow abstractions with mutable state and workset semantics for incremental computation.
• To demonstrate that dataflow systems can match specialized systems in performance without sacrificing expressiveness or transparency.

Proposed method

• Extends dataflow systems with incremental iterations, where each iteration updates only changed parts of the state using a workset of changed elements.
• Introduces a programming model that separates state mutation from message passing, enabling fine-grained control over updates.
• Uses a workset-based approach to track only the elements that need to be processed in each iteration, reducing redundant computation.
• Employs a dataflow optimizer to push down operations and minimize data movement across iterations.
• Supports asynchronous microstep execution under well-defined conditions, improving convergence speed for sparse algorithms.
• Implements the approach in Stratosphere, a parallel dataflow system, and compiles incremental iterations into optimized dataflow plans.

Experimental results

Research questions

• RQ1 Can incremental iterations be efficiently integrated into a general-purpose dataflow system to improve performance for iterative algorithms?
• RQ2 To what extent can dataflow systems exploit sparse computational dependencies inherent in graph and machine learning algorithms?
• RQ3 How does the performance of a dataflow system with incremental iterations compare to specialized systems like Pregel or GraphLab?
• RQ4 Can the unified dataflow abstraction maintain expressiveness while enabling optimizations for iterative workloads?
• RQ5 What are the conditions under which asynchronous microstep execution can be safely and efficiently applied in dataflow systems?

Key findings

• The proposed incremental iteration model achieves up to two orders of magnitude speedup over bulk iteration in dataflow systems for algorithms with sparse dependencies.
• The prototype implementation in Stratosphere achieves performance competitive with specialized systems like Pregel and GraphLab for graph algorithms such as PageRank and connected components.
• The integration of incremental iterations allows dataflow systems to express complex algorithms—such as adaptive PageRank—more naturally than Pregel, which couples activation and messaging.
• The system supports asynchronous microstep execution under well-defined conditions, enabling faster convergence without synchronization bottlenecks.
• The approach maintains a unified dataflow abstraction, reducing the need for multiple systems and orchestration frameworks in analytics pipelines.
• Compiler techniques can potentially transform fully iterative algorithms into incremental ones, suggesting further optimization potential.