[Paper Review] Efficient Detection of Complex Event Patterns Using Lazy Chain Automata

Ilya Kolchinsky, Assaf Schuster+1 more|arXiv (Cornell University)|Dec 15, 2016

Advanced Database Systems and Queries7 references3 citations

TL;DR

This paper proposes a lazy evaluation mechanism for Non-deterministic Finite Automata (NFA)-based Complex Event Processing (CEP) that reorders event processing by frequency to reduce redundant computation and memory usage. By processing rare events first using Chain NFA and Tree NFA topologies, the approach achieves up to two orders of magnitude improvement in throughput and significant memory reduction on real stock trading data.

ABSTRACT

Complex Event Processing (CEP) is an emerging field with important applications in many areas. CEP systems collect events arriving from input data streams and use them to infer more complex events according to predefined patterns. The Non-deterministic Finite Automaton (NFA) is one of the most popular mechanisms on which such systems are based. During the event detection process, NFAs incrementally extend previously observed partial matches until a full match for the query is found. As a result, each arriving event needs to be processed to determine whether a new partial match is to be initiated or an existing one extended. This method may be highly inefficient when many of the events do not result in output matches. We present a lazy evaluation mechanism that defers processing of frequent event types and stores them internally upon arrival. Events are then matched in ascending order of frequency, thus minimizing potentially redundant computations. We introduce a Lazy Chain NFA, which utilizes the above principle, and does not depend on the underlying pattern structure. An algorithm for constructing a Lazy Chain NFA for common pattern types is presented, including conjunction, negation and iteration. Finally, we experimentally evaluate our mechanism on real-world stock trading data. The results demonstrate a performance gain of two orders of magnitude over traditional NFA-based approaches, with significantly reduced memory resource requirements.

Motivation & Objective

• Address the inefficiency of traditional NFA-based CEP systems that maintain excessive partial matches when event arrival rates vary widely.
• Reduce memory and computational overhead in complex event processing by deferring processing of frequent events until rarer, more selective events are handled first.
• Develop a general-purpose mechanism applicable to complex patterns involving conjunction, negation, and iteration, not limited to simple sequences.
• Provide formal correctness proofs for the proposed lazy evaluation mechanism across diverse pattern types, filling a gap in prior work.
• Enable adaptive and frequency-aware processing without prior knowledge of event frequencies via the Tree NFA extension.

Proposed method

• Introduce a lazy evaluation model that stores incoming events in an input buffer and defers processing until necessary, based on event frequency.
• Design the Chain NFA topology, which requires pre-defined event frequency ordering and processes events in ascending order of frequency to minimize active partial matches.
• Construct a Tree NFA that dynamically computes the frequency order on-the-fly, enabling adaptation to unknown or changing event arrival rates.
• Provide a systematic algorithm for compiling common pattern types—such as sequences, conjunctions, negations, and iterations—into Chain NFA and Tree NFA structures.
• Ensure correctness of the matching process under the skip-till-any-match event selection strategy, allowing unlimited reuse of primitive events across matches.
• Leverage the topological structure of NFAs to delay computation on frequent events, reducing the number of active state instances and improving runtime efficiency.

Experimental results

Research questions

• RQ1 Can processing events in ascending order of frequency reduce the number of active partial matches in NFA-based CEP systems, thereby improving performance and reducing memory usage?
• RQ2 How can a lazy evaluation mechanism be formally integrated into NFA-based CEP to support complex patterns including conjunction, negation, and iteration?
• RQ3 What are the performance and memory trade-offs of using a pre-defined frequency order (Chain NFA) versus an adaptive, on-the-fly frequency computation (Tree NFA)?
• RQ4 Is it possible to achieve both high throughput and low memory consumption in CEP systems by deferring processing of frequent events while maintaining correctness for all possible matches?
• RQ5 How does the proposed lazy NFA model compare to traditional eager NFA evaluation in real-world workloads, particularly under skewed event arrival distributions?

Key findings

• The proposed lazy evaluation mechanism achieves a performance gain of up to two orders of magnitude (100x) in throughput compared to traditional eager NFA-based CEP systems on real stock trading data.
• Memory consumption is significantly reduced due to fewer active partial matches being maintained at runtime, especially when processing high-frequency events.
• The Chain NFA, which relies on pre-defined event frequency ordering, enables efficient processing by deferring frequent events and minimizing redundant state extensions.
• The Tree NFA dynamically computes the optimal processing order at runtime, making it suitable for environments where event frequencies are unknown or variable.
• The correctness of the lazy evaluation mechanism is formally proven for all standard CEP pattern types, including sequences, conjunctions, negations, and iterations.
• Experimental evaluation confirms that both Chain NFA and Tree NFA outperform eager NFA evaluation in terms of runtime complexity, memory usage, and overall system efficiency.