[Paper Review] Redesigning pattern mining algorithms for supercomputers

Kazuki Yoshizoe, Aika Terada+1 more|arXiv (Cornell University)|Oct 27, 2015

Data Mining Algorithms and Applications8 references3 citations

TL;DR

This paper presents a highly scalable, distributed-memory parallel algorithm for closed pattern mining tailored for supercomputers, leveraging a hypercube-structured communication model with multiple distributed stacks to balance workloads and minimize communication bottlenecks. The approach achieves up to 1175-fold speedup on 1200 cores when mining statistically significant mutation patterns from large-scale personal genome data, outperforming naive parallelization and single-core baselines on dense datasets.

ABSTRACT

Upcoming many core processors are expected to employ a distributed memory architecture similar to currently available supercomputers, but parallel pattern mining algorithms amenable to the architecture are not comprehensively studied. We present a novel closed pattern mining algorithm with a well-engineered communication protocol, and generalize it to find statistically significant patterns from personal genome data. For distributing communication evenly, it employs global load balancing with multiple stacks distributed on a set of cores organized as a hypercube with random edges. Our algorithm achieved up to 1175-fold speedup by using 1200 cores for solving a problem with 11,914 items and 697 transactions, while the naive approach of separating the search space failed completely.

Motivation & Objective

• To address the lack of scalable, distributed-memory-optimized pattern mining algorithms for upcoming many-core supercomputing architectures.
• To enable efficient mining of statistically significant patterns from dense, large-scale personal genome data, such as mutation profiles from HapMap and MCF7 datasets.
• To overcome communication bottlenecks in parallel search by distributing stacks across a hypercube with random edges for global load balancing.
• To integrate a multiple testing procedure (LAMP) with parallel pattern mining to ensure correct family-wise error rate control.
• To demonstrate superior scalability and performance on compute-intensive, high-density pattern mining workloads compared to naive or shared-memory approaches.

Proposed method

• The algorithm is built on the LCM (Linear time Closed itemset Miner) framework but redesigned for distributed memory by distributing multiple search stacks across a hypercube topology of compute nodes.
• It employs a global load balancing strategy where each core maintains a local stack and dynamically pulls work from neighboring cores via a randomized edge structure to balance communication and computation.
• The communication protocol uses a hypercube with random edges to evenly distribute message traffic and avoid hotspots, ensuring efficient parallelization of depth-first search on unbalanced trees.
• Statistical significance is assessed using the LAMP (Limitless-arity Multiple Testing Procedure) framework, applying Bonferroni-like corrections with Tarone’s P-value bounds to control family-wise error rate.
• The algorithm integrates depth-first search with dynamic work stealing and supports pruning via support counting, enabling efficient exploration of large search spaces.
• The implementation is optimized for high-density databases, with performance tuned for large itemsets and high transaction counts, as seen in personal genome datasets.

Experimental results

Research questions

• RQ1 Can a distributed-memory parallel pattern mining algorithm achieve high scalability and load balance on supercomputers with thousands of cores?
• RQ2 How effective is a hypercube-based communication model with random edges in reducing communication bottlenecks during parallel pattern mining?
• RQ3 To what extent can the integration of LAMP’s multiple testing correction improve the statistical reliability of mined patterns in large-scale genomics data?
• RQ4 How does the proposed algorithm compare to single-core and naive parallelized implementations in terms of speedup and scalability on dense, high-dimensional datasets?
• RQ5 Can the algorithm efficiently handle the computational demands of mining statistically significant mutation patterns from real-world personal genome data?

Key findings

• The proposed algorithm achieved up to 1175-fold speedup when using 1200 cores on a dataset with 11,914 items and 697 transactions, significantly outperforming naive parallelization.
• The algorithm successfully scaled to 1200 cores, with speedup increasing on more complex, dense problems, demonstrating strong weak scaling behavior.
• For the HapMap dominant 20 dataset, statistically significant itemsets with up to 8 items were discovered in under 20 seconds, a task infeasible with brute-force methods.
• The third phase of the pipeline, which computes P-values, took less than 10 ms per itemset, indicating minimal overhead in the statistical correction step.
• While single-core LAMP2 outperformed the parallel implementation on sparse datasets, the proposed method showed superior performance on dense, large-scale datasets, especially with increasing core counts.
• The hypercube-based communication model effectively balanced workloads and prevented communication hotspots, enabling efficient parallelization of depth-first search on unbalanced trees.