[Paper Review] Two simple full-text indexes based on the suffix array

Szymon Grabowski, Marcin Raniszewski|arXiv (Cornell University)|May 22, 2014

Algorithms and Data Compression7 references3 citations

TL;DR

This paper proposes two suffix array-based full-text indexes: SA-hash, which accelerates pattern searches by using a hash table to narrow the initial search interval, and FBCSA, a compact, block-based suffix array variant optimized for fast single-cell access. SA-hash achieves 2–3× speedup over standard suffix arrays with 0.2n–1.1n bytes extra space, while FBCSA offers several times faster single-cell access than related compact indexes at comparable compression, though it underperforms for bulk extractions.

ABSTRACT

We propose two suffix array inspired full-text indexes. One, called SA-hash, augments the suffix array with a hash table to speed up pattern searches due to significantly narrowed search interval before the binary search phase. The other, called FBCSA, is a compact data structure, similar to M{ä}kinen's compact suffix array, but working on fixed sized blocks. Experiments on the Pizza~\&~Chili 200\,MB datasets show that SA-hash is about 2--3 times faster in pattern searches (counts) than the standard suffix array, for the price of requiring $0.2n-1.1n$ bytes of extra space, where $n$ is the text length, and setting a minimum pattern length. FBCSA is relatively fast in single cell accesses (a few times faster than related indexes at about the same or better compression), but not competitive if many consecutive cells are to be extracted. Still, for the task of extracting, e.g., 10 successive cells its time-space relation remains attractive.

Motivation & Objective

• To design faster full-text indexes that prioritize search performance over memory compression, especially for applications where speed is critical.
• To explore practical space-time tradeoffs by enhancing the standard suffix array with auxiliary data structures.
• To present a compact, block-aligned variant of Mäkinen’s compact suffix array that supports efficient random access to individual SA cells.
• To evaluate the performance tradeoffs between speed, space, and access patterns (single vs. consecutive cell access) in suffix array-based indexes.

Proposed method

• SA-hash augments the suffix array with a hash table that stores the start and end indices of SA intervals for all prefixes of length k (e.g., k=3–4), enabling rapid narrowing of the binary search range.
• The hash table is built over all possible k-length prefixes of the text, with each entry mapping a prefix to the interval [l, r] of suffix array indices where suffixes starting with that prefix are located.
• FBCSA partitions the suffix array into fixed-size blocks (bs = 32 or 64), storing only sampled SA values and using a compressed layout to achieve space efficiency.
• For each block, FBCSA stores a small number of explicitly stored SA values (via sampling step ss) and uses a lookup table to reconstruct intermediate values, enabling fast random access.
• The structure uses a compact representation that aligns with 32-bit boundaries, improving memory access efficiency and reducing space usage.
• Both indexes are evaluated on Pizza & Chili 200 MB datasets, with performance measured in pattern search (count), single-cell access, and consecutive cell extraction times.

Experimental results

Research questions

• RQ1 Can hashing be effectively used to accelerate pattern search in suffix array-based full-text indexes by reducing the initial search interval?
• RQ2 How does the performance of a block-based, compact suffix array variant (FBCSA) compare to existing compact indexes in terms of single-cell access speed and space usage?
• RQ3 What is the tradeoff between space overhead and search speed when using a hash table to precompute interval boundaries in suffix array queries?
• RQ4 How does the sampling step (ss) and block size (bs) affect the performance and space usage of FBCSA in random and consecutive cell access patterns?
• RQ5 Can FBCSA achieve competitive performance for small-range extractions (e.g., 5–10 consecutive cells) despite not being optimized for bulk access?

Key findings

• SA-hash achieves 2–3 times faster pattern search (count queries) than the standard suffix array, with a space overhead of 0.2n to 1.1n bytes, depending on the dataset and k-value.
• The performance gain in SA-hash is attributed to significantly narrowed search intervals before binary search, reducing the number of comparisons needed.
• FBCSA is several times faster than MakCSA and LCSA/Psi in single-cell SA access, achieving competitive performance at similar or better compression ratios.
• For consecutive cell access (e.g., 5–10 cells), FBCSA remains competitive due to sub-linear time growth, especially with smaller sampling steps (ss), though it is outperformed by specialized indexes for large-range extractions.
• On count queries, FBCSA variants are competitive with MakCSA on DNA and English datasets but significantly slower on XML and sources datasets, particularly under logarithmic scaling.
• The space overhead of lookup tables (LUT2, LUT3) in SA-hash is reduced by a factor of ~4 for 200 MB datasets, suggesting scalability for larger texts.