[Paper Review] Derivative Based Extended Regular Expression Matching Supporting Intersection, Complement and Lookarounds

Ian Erik Varatalu, Margus Veanes+1 more|arXiv (Cornell University)|Sep 25, 2023

Software Engineering Research4 citations

TL;DR

This paper presents a symbolic derivatives-based regular expression engine that extends standard regex with support for intersection, complement, and both positive and negative lookarounds (lookaheads and lookbacks), enabling precise, efficient, and semantically consistent matching. The approach leverages an effective Boolean algebra over match sets to allow optimization via Boolean rewrite rules, demonstrating feasibility for problems intractable with existing engines due to factorial blowup in backtracking approaches.

ABSTRACT

Regular expressions are widely used in software. Various regular expression engines support different combinations of extensions to classical regular constructs such as Kleene star, concatenation, nondeterministic choice (union in terms of match semantics). The extensions include e.g. anchors, lookarounds, counters, backreferences. The properties of combinations of such extensions have been subject of active recent research. In the current paper we present a symbolic derivatives based approach to finding matches to regular expressions that, in addition to the classical regular constructs, also support complement, intersection and lookarounds (both negative and positive lookaheads and lookbacks). The theory of computing symbolic derivatives and determining nullability given an input string is presented that shows that such a combination of extensions yields a match semantics that corresponds to an effective Boolean algebra, which in turn opens up possibilities of applying various Boolean logic rewrite rules to optimize the search for matches. In addition to the theoretical framework we present an implementation of the combination of extensions to demonstrate the efficacy of the approach accompanied with practical examples.

Motivation & Objective

• To develop a regular expression engine that supports intersection, complement, and lookarounds within a symbolic derivatives framework.
• To demonstrate that the combination of these extensions yields a match semantics closed under Boolean operations, forming an effective Boolean algebra.
• To enable practical, high-performance regex matching for problems previously infeasible with backtracking or standard nonbacktracking engines.
• To prove and implement the conjecture that all context-sensitive anchors (e.g., word boundaries) can be expressed using lookarounds.
• To provide a foundation for optimization through Boolean rewrite rules and efficient caching techniques in regex matching.

Proposed method

• The paper extends the .NET 7 nonbacktracking regex engine with symbolic derivatives to support intersection (&), complement (~), and lookarounds (?=, ?!, ?<=, ?<!).
• It introduces a theoretical framework based on symbolic derivatives that ensures correctness and determinism in match semantics.
• The match set semantics are shown to form an effective Boolean algebra, enabling systematic application of Boolean rewrite rules for optimization.
• The implementation uses a derivative-based evaluation engine that computes derivatives of regexes with respect to input characters, tracking match positions and predicates.
• Lookarounds are handled by evaluating the lookahead/lookbehind condition at each position without consuming input, using a derived predicate to determine match validity.
• Caching techniques are proposed for lookarounds by storing and invalidating predicates (e.g., ψ for character classes) when encountering disqualifying characters like newline.

Figure 2 . Time taken to find a matching paragraph containing all required substrings in any order in a 9kB sample text.

Experimental results

Research questions

• RQ1 Can intersection, complement, and lookarounds be meaningfully combined in a symbolic derivatives-based regex engine without sacrificing determinism or performance?
• RQ2 Does the combination of these extensions yield a match semantics closed under Boolean operations, forming an effective Boolean algebra?
• RQ3 Can such an extended engine solve problems that cause factorial blowup in backtracking engines, such as locating substrings with matches in arbitrary order?
• RQ4 Can all context-sensitive anchors (e.g., word boundaries) be expressed using only lookarounds, as conjectured in prior work?
• RQ5 To what extent can Boolean rewrite rules and caching techniques improve performance in regex matching with complex extensions?

Key findings

• The extended regex engine successfully handles problems involving arbitrary-order match location that cause factorial blowup in .NET 7’s backtracking and nonbacktracking engines.
• The match semantics of the extended regex system form an effective Boolean algebra, enabling systematic optimization via Boolean rewrite rules.
• All context-sensitive anchors, including word boundaries and line anchors, can be expressed using only lookarounds, and this is formally proven and implemented.
• The engine demonstrates linear time complexity for short-context lookarounds (e.g., ^, $, \b), while longer lookarounds (e.g., (?<=a.*)b) can be optimized via predicate caching to avoid quadratic behavior.
• Laziness in regexes can be emulated using complement and intersection, such as a[^b\n]*b or (a.*)&(~(.*b.*)b), providing equivalent semantics without lazy quantifiers.
• The implementation reuses the .NET 7 nonbacktracking engine backend and extends it with new operators, maintaining performance and correctness while enabling new use cases.

Figure 3 . Time taken to find a matching paragraph containing all required substrings in any order in the full 20MB sample text.