[Paper Review] Engineering Small Space Dictionary Matching

Shoshana Marcus Dina Sokol|arXiv (Cornell University)|Jan 28, 2013

Algorithms and Data Compression10 references3 citations

TL;DR

This paper presents a space-efficient dictionary matching algorithm that leverages the compressed suffix tree as its core data structure, enabling linear-time pattern matching with working space proportional to the empirical entropy of the dictionary. The key contribution is a succinct framework for constant-time lowest marked ancestor queries on compressed suffix trees, which enhances efficiency without sacrificing space savings, validated through experiments on DNA and English text with up to 5.4MB space usage and near-linear search times.

ABSTRACT

The dictionary matching problem is to locate occurrences of any pattern among a set of patterns in a given text. Massive data sets abound and at the same time, there are many settings in which working space is extremely limited. We introduce dictionary matching software for the space-constrained environment whose running time is close to linear. We use the compressed suffix tree as the underlying data structure of our algorithm, thus, the working space of our algorithm is proportional to the optimal compression of the dictionary. We also contribute a succinct tool for performing constant-time lowest marked ancestor queries on a tree that is succinctly encoded as a sequence of balanced parentheses, with linear time preprocessing of the tree. This tool should be useful in many other applications. Our source code is available at http://www.sci.brooklyn.cuny.edu/~sokol/dictmatch.html

Motivation & Objective

• To address the challenge of dictionary matching in space-constrained environments such as mobile and satellite devices, where data growth outpaces available storage.
• To bridge the gap between theoretically optimal succinct dictionary matching algorithms and practical software implementations.
• To develop a practical, linear-time dictionary matching system that operates within space proportional to the empirical entropy of the dictionary.
• To design and implement a succinct data structure for constant-time lowest marked ancestor queries on compressed suffix trees, reusable across other applications.

Proposed method

• The algorithm uses the compressed suffix tree (CST) of Sadakane as the underlying data structure, which supports efficient space usage through compressed representations of the suffix array and LCP array.
• The method integrates a bit vector and a sequence of balanced parentheses to succinctly encode the tree structure, enabling efficient navigation and query operations.
• A novel succinct framework is introduced for answering lowest marked ancestor queries in constant time, using preprocessed bit vectors and balanced parentheses sequences.
• The implementation leverages the Succinct Data Structures Library and supports multiple compressed suffix array and LCP array variants (e.g., csa_sada, csa_wt, lcp_dac, lcp_support_tree2) for performance tuning.
• The system is evaluated using two real-world datasets: 5MB of human genome DNA and 5MB of English text, with dictionaries of 2.5–3MB.
• Preprocessing and search times are measured across multiple CST configurations to assess time-space trade-offs.

Experimental results

Research questions

• RQ1 Can a succinct dictionary matching algorithm be implemented in practice with near-linear running time and space proportional to the empirical entropy of the dictionary?
• RQ2 How effective is the proposed lowest marked ancestor query structure in maintaining constant-time performance on compressed suffix trees?
• RQ3 What is the practical time-space trade-off when using different compressed representations of the suffix array and LCP array in a dictionary matching system?
• RQ4 To what extent does the compressed suffix tree reduce memory usage compared to uncompressed alternatives, without significantly degrading search performance?

Key findings

• The compressed suffix tree configuration using csa_wt and lcp_support_tree2 achieved the smallest memory footprint of 5.4MB for DNA data and 5.0MB for English text, while maintaining near-linear search times.
• Search times across all compressed configurations were within 1% of the uncompressed baseline (2.05 hours for DNA, 3.63 hours for English), indicating negligible performance degradation.
• Preprocessing times ranged from 7 to 30 seconds, with the most space-efficient configurations requiring slightly longer setup, but still within practical limits.
• The lowest marked ancestor query framework enabled constant-time operations after linear-time preprocessing, demonstrating its efficiency and reusability beyond dictionary matching.
• The system achieved space savings of up to 80% compared to uncompressed suffix trees (e.g., 26.7MB vs. 5.4MB for DNA), confirming the effectiveness of compression.
• The results validate that succinct data structures can be used effectively in practical dictionary matching systems, closing the gap between theory and implementation.