[Paper Review] Automated Synthesis of Divide and Conquer Parallelism

Azadeh Farzan, Victor Nicolet|arXiv (Cornell University)|Jan 29, 2017

Parallel Computing and Optimization Techniques31 references3 citations

TL;DR

This paper presents an automated technique for synthesizing divide-and-conquer parallelism in sequential code by automatically generating correct join operations and, when necessary, extending the original code with auxiliary accumulators to enable parallelization. The approach uses syntax-guided synthesis with machine-checked correctness proofs, achieving near-ideal speedups on diverse benchmarks without runtime overhead.

ABSTRACT

This paper focuses on automated synthesis of divide-and-conquer parallelism, which is a common parallel programming skeleton supported by many cross-platform multithreaded libraries. The challenges of producing (manually or automatically) a correct divide-and-conquer parallel program from a given sequential code are two-fold: (1) assuming that individual worker threads execute a code identical to the sequential code, the programmer has to provide the extra code for dividing the tasks and combining the computation results, and (2) sometimes, the sequential code may not be usable as is, and may need to be modified by the programmer. We address both challenges in this paper. We present an automated synthesis technique for the case where no modifications to the sequential code are required, and we propose an algorithm for modifying the sequential code to make it suitable for parallelization when some modification is necessary. The paper presents theoretical results for when this {\em modification} is efficiently possible, and experimental evaluation of the technique and the quality of the produced parallel programs.

Motivation & Objective

• To automate the synthesis of correct join operations for divide-and-conquer parallelism in sequential code.
• To address the challenge of parallelizing loops that lack a natural join operation by automatically extending the code with auxiliary accumulators.
• To provide theoretical foundations for when such extensions are efficiently possible and can be automatically discovered.
• To generate machine-checked proofs of correctness for all synthesized join operations, ensuring correctness across all inputs.
• To evaluate the performance and scalability of the synthesized parallel programs on real-world benchmarks.

Proposed method

• Using syntax-guided synthesis (SGS) to search for correct join operators in a constrained but expressive search space.
• Defining a formal framework to automatically extend sequential loops with auxiliary accumulators to enable the existence of a valid join operation.
• Leveraging the third homomorphism theorem and related algebraic principles to guide the synthesis of join functions.
• Integrating automated proof generation to verify correctness of synthesized join operations across the entire input domain.
• Designing an algorithm that balances correctness, efficiency, and low overhead in the extended code.
• Applying the method to a range of benchmarks, including reductions and non-associative operations like is-sorted and length.

Experimental results

Research questions

• RQ1 Can a correct join operator be automatically synthesized for divide-and-conquer parallelism when one exists?
• RQ2 Under what conditions can a non-parallelizable loop be automatically extended to support efficient divide-and-conquer parallelization?
• RQ3 Is it possible to automatically generate machine-checked proofs of correctness for synthesized join operations?
• RQ4 Can the overhead of code extension be kept low while ensuring performance gains from parallelization?
• RQ5 How does the performance of the synthesized parallel programs compare to sequential and hand-optimized versions?

Key findings

• The proposed method successfully synthesizes correct join operations for all tested benchmarks, including complex cases like is-sorted and length, where the join is not immediately obvious.
• The algorithm automatically generates code extensions with auxiliary accumulators that enable parallelization where it was previously impossible.
• All synthesized join operations are accompanied by machine-checked correctness proofs, ensuring correctness across the entire input domain.
• The technique achieves near-ideal speedups on average, with a mean speedup of 9.8 and a standard deviation of 0.04, indicating high consistency and scalability.
• The approach outperforms prior work such as SYMPA and [33], as it requires only a single sequential implementation as input, unlike methods requiring left- and rightward traversals.
• The method is effective even for functions that are not associative by default, such as length and is-sorted, by automatically inferring the necessary auxiliary information.