[Paper Review] Parallel Quicksort without Pairwise Element Exchange

Jesper Larsson Träff|arXiv (Cornell University)|Apr 20, 2018

Parallel Computing and Optimization Techniques13 references3 citations

TL;DR

This paper proposes exchange-free parallel Quicksort variants that eliminate pairwise element exchanges during recursion by instead distributing only pivots, reducing total communication volume from $O(n\log p)$ to $O(n)$. The method uses MPI collective operations for pivot distribution and single redistribution at recursion bottom, achieving up to 650x speed-up on 1024 processes and 1.3x faster weak scaling than standard variants when elements per process exceed 500,000.

ABSTRACT

Quicksort is an instructive classroom approach to parallel sorting on distributed memory parallel computers with many opportunities for illustrating specific implementation alternatives and tradeoffs with common communication interfaces like MPI. The (two) standard distributed memory Quicksort implementations exchange partitioned data elements at each level of the Quicksort recursion. In this note, we show that this is not necessary: It suffices to distribute only the chosen pivots, and postpone element redistribution to the bottom of the recursion. This reduces the total volume of data exchanged from $O(n\log p)$ to $O(n)$, $n$ being the total number of elements to be sorted and $p$ a power-of-two number of processors, by trading off against a total of $O(p)$ additional pivot element distributions. Based on this observation, we describe new, \emph{exchange-free} implementation variants of parallel Quicksort and of Wagar's HyperQuicksort. We have implemented the discussed four different Quicksort variations in MPI, and show that with good pivot selection, Quicksort without pairwise element exchange can be significantly faster than standard implementations on moderately large problems.

Motivation & Objective

• To eliminate pairwise element exchanges in distributed memory parallel Quicksort to reduce communication volume and improve scalability.
• To explore the trade-off between increased pivot distribution and reduced element exchange in parallel sorting.
• To design and implement exchange-free variants of Quicksort and HyperQuicksort using only MPI collective operations.
• To evaluate performance on large-scale clusters and compare with standard exchange-based implementations.
• To combine exchange-based and exchange-free variants for smooth scaling across different problem sizes.

Proposed method

• Replace recursive pairwise element exchanges with global pivot distribution using MPI_Alltoallv and MPI_Allreduce for pivot selection.
• Postpone actual element redistribution until the bottom of the recursion, using only pivot information during higher-level steps.
• Use a single redistribution step at the recursion base, where each process sorts its local set based on pivot boundaries.
• Implement two variants: one for standard Quicksort and one for Wagar’s HyperQuicksort, both exchange-free.
• Combine standard and exchange-free variants by switching to the exchange-free mode when elements per process exceed a threshold (e.g., c=1500).
• Use MPI processes ranked 0 to p−1, with p a power of two, and assume initial load balance with n/p elements per process.

Experimental results

Research questions

• RQ1 Can pairwise element exchanges in distributed memory Quicksort be eliminated without degrading performance?
• RQ2 What is the communication cost trade-off between exchanging elements and distributing pivots in recursive Quicksort?
• RQ3 How does the performance of exchange-free Quicksort compare to standard exchange-based Quicksort on large-scale clusters?
• RQ4 Can a hybrid approach combining both exchange-based and exchange-free variants achieve better scalability across varying problem sizes?
• RQ5 What threshold of elements per process makes the exchange-free variant more efficient than the standard one?

Key findings

• On 1024 processes, the exchange-free Quicksort achieved a speed-up of 650 when sorting 10^8 floating-point numbers, significantly outperforming the standard variant.
• In weak scaling with 8192 processes, exchange-free variants were consistently 1.3× faster than standard variants when elements per process exceeded 500,000.
• The combined Quicksort variant never underperformed either standard or exchange-free version and achieved a speed-up exceeding 1000 on 4096 processes.
• For small elements per process (n/p < 100,000), the performance of exchange-free variants degraded due to high-latency MPI_Alltoallv and MPI_Allreduce operations.
• The exchange-free approach reduced total communication volume from $O(n\log p)$ to $O(n)$, with a trade-off of $O(p)$ additional pivot distributions.
• Results were robust across different data types (doubles, integers) and input distributions (random permutations), showing consistent performance gains for moderate to large problem sizes.