[Paper Review] Online Pattern Matching for String Edit Distance with Moves

Yoshimasa Takabatake, Yasuo Tabei+1 more|arXiv (Cornell University)|Aug 3, 2014

Algorithms and Data Compression18 references4 citations

TL;DR

This paper proposes Online Edit Sensitive Parsing (OESP), an online algorithm for approximate string edit distance with moves (EDM) that processes streaming text in real time. By building a succinct parse tree representation using post-order unary degree sequences (POUDS) and maintaining characteristic vectors for online L1 distance computation, OESP achieves O(lg²N)-approximation of EDM with sublinear space and near-linear time complexity, enabling efficient pattern matching on large-scale streaming data.

ABSTRACT

Edit distance with moves (EDM) is a string-to-string distance measure that includes substring moves in addition to ordinal editing operations to turn one string to the other. Although optimizing EDM is intractable, it has many applications especially in error detections. Edit sensitive parsing (ESP) is an efficient parsing algorithm that guarantees an upper bound of parsing discrepancies between different appearances of the same substrings in a string. ESP can be used for computing an approximate EDM as the L1 distance between characteristic vectors built by node labels in parsing trees. However, ESP is not applicable to a streaming text data where a whole text is unknown in advance. We present an online ESP (OESP) that enables an online pattern matching for EDM. OESP builds a parse tree for a streaming text and computes the L1 distance between characteristic vectors in an online manner. For the space-efficient computation of EDM, OESP directly encodes the parse tree into a succinct representation by leveraging the idea behind recent results of a dynamic succinct tree. We experimentally test OESP on the ability to compute EDM in an online manner on benchmark datasets, and we show OESP's efficiency.

Motivation & Objective

• To address the challenge of computing edit distance with moves (EDM) on streaming text, where the full string is unknown in advance.
• To extend offline Edit Sensitive Parsing (ESP) to an online setting for real-time pattern matching in applications like web mining and bioinformatics.
• To minimize space usage while maintaining a provable approximation ratio for EDM in an online streaming environment.
• To design a succinct data structure that supports dynamic parsing and efficient distance computation on the fly.

Proposed method

• OESP incrementally builds a parse tree for streaming text using a left-aligned parsing strategy similar to offline ESP.
• It maintains characteristic vectors of node labels in the parse tree to compute L1 distance between query and substrings in an online manner.
• A novel succinct tree representation called post-order unary degree sequence (POUDS) is introduced to compress the parse tree using post-order traversal and unary encoding.
• The method leverages dynamic succinct tree techniques and online grammar compression to support space-efficient, incremental updates.
• It uses a parameterized hash table with parameter α to balance time and space efficiency in the online parsing process.
• The algorithm computes the L1 distance between characteristic vectors of substrings and a query string in real time, enabling online pattern matching.

Experimental results

Research questions

• RQ1 Can edit distance with moves (EDM) be approximated efficiently in an online streaming setting where the full text is not available upfront?
• RQ2 What is the approximation ratio of an online algorithm for EDM that maintains a succinct data structure for dynamic parsing?
• RQ3 How can a parse tree be incrementally constructed and compressed in real time to support online pattern matching with low space overhead?
• RQ4 Can the characteristic vector approach used in offline ESP be adapted for online computation with provable bounds?

Key findings

• OESP achieves an O(lg²N)-approximation ratio for edit distance with moves, which is a provably bounded approximation of the exact EDM.
• The algorithm runs in O(N lgN lg n / (α lg lg n)) expected time, demonstrating near-linear scalability with input size.
• The space complexity is O(n(α+1) lg(n+σ) + n lg(αn) + 5n + o(n)) bits, where n is the number of variables in the grammar, showing sublinear dependence on text length.
• Experiments on 200MB DNA and English text streams showed that OESP’s working space grows linearly with text length, with dictionary and bitstring representations being significantly smaller than hash tables.
• The space usage for the dictionary was 115 MB on DNA and 121 MB on English data, while the hash table consumed 368 MB and 382 MB respectively, confirming space efficiency.
• Figure 6 shows that the number of substrings within a given EDM threshold increases dramatically at certain thresholds, indicating OESP’s practical utility in identifying similar patterns in streaming data.