[Paper Review] ERA: Efficient Serial and Parallel Suffix Tree Construction for Very Long Strings

Essam Mansour, Amin Allam+2 more|arXiv (Cornell University)|Sep 30, 2011

Algorithms and Data Compression8 references4 citations

TL;DR

This paper proposes ERa, a disk-based suffix tree construction algorithm that dynamically partitions string processing both vertically and horizontally to minimize I/O and optimize memory usage. It achieves sub-20-minute construction of the human genome on a standard 8-core desktop with 16GB RAM—surpassing the performance of prior methods that require supercomputers with thousands of cores.

ABSTRACT

The suffix tree is a data structure for indexing strings. It is used in a variety of applications such as bioinformatics, time series analysis, clustering, text editing and data compression. However, when the string and the resulting suffix tree are too large to fit into the main memory, most existing construction algorithms become very inefficient. This paper presents a disk-based suffix tree construction method, called Elastic Range (ERa), which works efficiently with very long strings that are much larger than the available memory. ERa partitions the tree construction process horizontally and vertically and minimizes I/Os by dynamically adjusting the horizontal partitions independently for each vertical partition, based on the evolving shape of the tree and the available memory. Where appropriate, ERa also groups vertical partitions together to amortize the I/O cost. We developed a serial version; a parallel version for shared-memory and shared-disk multi-core systems; and a parallel version for shared-nothing architectures. ERa indexes the entire human genome in 19 minutes on an ordinary desktop computer. For comparison, the fastest existing method needs 15 minutes using 1024 CPUs on an IBM BlueGene supercomputer.

Motivation & Objective

• To address the inefficiency of existing suffix tree construction algorithms when handling very long strings that exceed main memory capacity.
• To design a method that minimizes I/O operations and dynamically adapts to evolving tree structures and available memory.
• To enable efficient serial and parallel suffix tree construction on commodity hardware, including shared-memory, shared-disk, and shared-nothing architectures.
• To achieve high performance on real-world, large-scale datasets such as the human genome and financial time series.

Proposed method

• ERa partitions the suffix tree construction process vertically into independent sub-trees, each small enough to fit in main memory.
• It further divides each sub-tree horizontally into partitions that can be processed with a single sequential scan of the input string.
• The algorithm dynamically adjusts horizontal partition sizes based on the evolving shape of the tree and available memory to maximize utilization.
• Vertical partitions are grouped together to amortize I/O costs across multiple sub-trees.
• The method supports serial execution and multiple parallel architectures, including shared-memory, shared-disk, and shared-nothing clusters.
• It uses sequential I/O access to the input string, which is significantly faster than random I/O on modern disks.

Experimental results

Research questions

• RQ1 Can suffix tree construction for very long strings be made efficient when the data exceeds main memory capacity?
• RQ2 How can I/O overhead be minimized in disk-based suffix tree construction while maintaining high memory utilization?
• RQ3 Can a single algorithm efficiently support both serial and parallel execution on diverse architectures including desktops and clusters?
• RQ4 How does the algorithm scale with increasing string length and alphabet size compared to existing methods?
• RQ5 To what extent can dynamic partitioning and I/O amortization improve performance on real-world datasets like the human genome?

Key findings

• ERa constructs the entire human genome suffix tree in 19 minutes on an ordinary 8-core desktop with 16GB RAM, outperforming existing methods that require supercomputers.
• On a 16-node cluster with 4GB RAM per node, ERa completes construction in 8.3 minutes, the fastest reported time for this task.
• ERa is at least 50% faster than existing serial algorithms, with even greater gains on very long strings and large alphabets.
• In shared-nothing clusters, ERa shows a 2.5× performance advantage over WaveFront at 4096MBps input size, with a much slower rate of performance degradation.
• The algorithm achieves near-optimal speed-up in shared-memory and cluster environments, indicating excellent load balancing and scalability.
• ERa’s I/O cost is amortized through vertical partition grouping and dynamic horizontal partitioning, significantly reducing disk I/O overhead compared to prior methods.