[Paper Review] A Dichotomy for Regular Expression Membership Testing

Karl Bringmann, Allan Grønlund+1 more|arXiv (Cornell University)|Nov 3, 2016

Complexity and Algorithms in Graphs1 references4 citations

TL;DR

This paper establishes a complete dichotomy for regular expression membership testing under the Strong Exponential Time Hypothesis (SETH), proving that all bounded-depth homogeneous regular expression types are either solvable in almost-linear time or require quadratic time, with the Word Break problem being the sole exception. It presents a deterministic Õ(nm¹ᐟ³ + m) algorithm for Word Break and proves a matching conditional lower bound, establishing it as the only intermediate problem in this classification.

ABSTRACT

We study regular expression membership testing: Given a regular expression of size $m$ and a string of size $n$, decide whether the string is in the language described by the regular expression. Its classic $O(nm)$ algorithm is one of the big success stories of the 70s, which allowed pattern matching to develop into the standard tool that it is today. Many special cases of pattern matching have been studied that can be solved faster than in quadratic time. However, a systematic study of tractable cases was made possible only recently, with the first conditional lower bounds reported by Backurs and Indyk [FOCS'16]. Restricted to any "type" of homogeneous regular expressions of depth 2 or 3, they either presented a near-linear time algorithm or a quadratic conditional lower bound, with one exception known as the Word Break problem. In this paper we complete their work as follows: 1) We present two almost-linear time algorithms that generalize all known almost-linear time algorithms for special cases of regular expression membership testing. 2) We classify all types, except for the Word Break problem, into almost-linear time or quadratic time assuming the Strong Exponential Time Hypothesis. This extends the classification from depth 2 and 3 to any constant depth. 3) For the Word Break problem we give an improved $ ilde{O}(n m^{1/3} + m)$ algorithm. Surprisingly, we also prove a matching conditional lower bound for combinatorial algorithms. This establishes Word Break as the only intermediate problem. In total, we prove matching upper and lower bounds for any type of bounded-depth homogeneous regular expressions, which yields a full dichotomy for regular expression membership testing.

Motivation & Objective

• To complete the classification of bounded-depth homogeneous regular expression membership problems into almost-linear time or quadratic time under the Strong Exponential Time Hypothesis (SETH).
• To resolve the open status of the Word Break problem by providing a faster deterministic algorithm and proving a matching conditional lower bound.
• To extend the dichotomy from depth 2 and 3 to any constant depth, closing gaps left by prior work.
• To establish that Word Break is the only problem in the intermediate complexity class between near-linear and quadratic time.

Proposed method

• Developed two new almost-linear time algorithms that generalize existing solutions for special cases of regular expression membership testing.
• Used fine-grained reductions to show that membership for any type t is either solvable in almost-linear time or hard under SETH, by reducing to known hard types like ∘⋆, ∘|∘, or ∘+∘.
• Introduced a novel deterministic algorithm for the Word Break problem running in Õ(nm¹ᐟ³ + m) time, improving upon the previous randomized Õ(nm¹ᐟ²⁻¹ᐟ¹⁸ + m) bound.
• Proved a matching conditional lower bound for combinatorial algorithms on Word Break, showing that no faster combinatorial algorithm exists unless SETH fails.
• Applied simplification rules to eliminate redundant types and reduce the problem space to minimal hard and easy cases.
• Constructed tree diagrams of all possible types to systematically classify membership problems by depth and operation sequence, marking each with algorithmic complexity or hardness.

Experimental results

Research questions

• RQ1 Is there a complete dichotomy for bounded-depth homogeneous regular expression membership testing under SETH, separating problems into almost-linear time or quadratic time complexity?
• RQ2 Can the Word Break problem be solved faster than the previously known randomized Õ(nm¹ᐟ²⁻¹ᐟ¹⁸ + m) bound, and is this improvement possible deterministically?
• RQ3 Does the Word Break problem represent the only intermediate case between near-linear and quadratic time complexity in regular expression membership testing?
• RQ4 Can the classification of membership problems be extended beyond depth 3 to any constant depth?
• RQ5 Is there a matching conditional lower bound for the Word Break problem that rules out faster combinatorial algorithms?

Key findings

• The Word Break problem admits a deterministic Õ(nm¹ᐟ³ + m) algorithm, improving upon the prior randomized Õ(nm¹ᐟ²⁻¹ᐟ¹⁸ + m) bound.
• A matching conditional lower bound for combinatorial algorithms on Word Break was proven, showing that no faster combinatorial algorithm exists unless the Strong Exponential Time Hypothesis (SETH) fails.
• All bounded-depth homogeneous regular expression membership problems are classified into two categories: either solvable in almost-linear time or requiring (nm)¹⁻ᵒ⁽¹⁾ time under SETH.
• The Word Break problem is the only problem in the intermediate complexity class, making it the sole exception to the dichotomy.
• The classification extends to all constant depths, completing the dichotomy for regular expression membership testing.
• The results are based on a systematic reduction framework and tree-based enumeration of all possible types, with simplification rules and hardness propagation via reductions.