[Paper Review] From Complex Event Processing to Simple Event Processing

Sylvain Hallé|arXiv (Cornell University)|Feb 26, 2017

Advanced Database Systems and Queries3 references7 citations

TL;DR

This paper proposes BeepBeep, a lightweight, extensible, and zero-configuration event stream processing engine that bridges the gap between low-level scripting and complex CEP systems. It introduces a modular processor-based architecture and a SQL-like query language (eSQL) to enable simple, composable, and extensible stream processing for diverse use cases, achieving reasonable performance for medium-duty workloads while supporting user-defined extensions via customizable palettes.

ABSTRACT

Many problems in Computer Science can be framed as the computation of queries over sequences, or "streams" of data units called events. The field of Complex Event Processing (CEP) relates to the techniques and tools developed to efficiently process these queries. However, most CEP systems developed so far have concentrated on relatively narrow types of queries, which consist of sliding windows, aggregation functions, and simple sequential patterns computed over events that have a fixed tuple structure. Many of them boast throughput, but in counterpart, they are difficult to setup and cumbersome to extend with user-defined elements. This paper describes a variety of use cases taken from real-world scenarios that present features seldom considered in classical CEP problems. It also provides a broad review of current solutions, that includes tools and techniques going beyond typical surveys on CEP. From a critical analysis of these solutions, design principles for a new type of event stream processing system are exposed. The paper proposes a simple, generic and extensible framework for the processing of event streams of diverse types; it describes in detail a stream processing engine, called BeepBeep, that implements these principles. BeepBeep's modular architecture, which borrows concepts from many other systems, is complemented with an extensible query language, called eSQL. The end result is an open, versatile, and reasonably efficient query engine that can be used in situations that go beyond the capabilities of existing systems.

Motivation & Objective

• To address the complexity and rigidity of existing Complex Event Processing (CEP) systems, which often require extensive configuration and lack extensibility.
• To identify a missing niche in event processing tools—between low-level scripts and heavyweight distributed platforms—by proposing a 'Simple Event Processing' solution.
• To design a system that supports rich query expressiveness, user-defined constructs, and seamless integration of domain-specific processing units.
• To enable users to define custom processors and extend the query language via dynamic 'palettes' without modifying core components.
• To demonstrate that a lightweight system can handle complex queries beyond basic windowing and aggregation, including temporal logic and pattern matching.

Proposed method

• The system is built around a modular processor architecture where each processor performs a specific transformation on event streams, enabling composable data pipelines.
• A declarative query language, eSQL, is designed for relational transparency, ensuring backward compatibility with SQL for common operations.
• Extensibility is achieved through 'palettes'—dynamically loadable packages that bundle custom processors and grammar extensions for eSQL.
• The computational model is formally specified to ensure semantic clarity and enable independent implementations.
• The engine supports both stateless and stateful processing, including complex event correlation across multiple streams and time windows.
• A GUI-based interface is planned to complement eSQL, allowing users to visually construct processing chains more intuitively than writing text-based queries.

Experimental results

Research questions

• RQ1 Can a lightweight, zero-configuration event processing system effectively handle complex queries beyond basic windowing and aggregation, such as temporal logic and pattern matching?
• RQ2 How can a query language be designed to be both simple and extensible, supporting user-defined constructs while maintaining compatibility with SQL-like syntax?
• RQ3 What architectural principles enable a system to be both modular and efficient enough for medium-duty stream processing tasks?
• RQ4 To what extent can a processor-based model support expressive, composable data processing chains without sacrificing performance?
• RQ5 How does such a system compare in practice to existing CEP platforms in terms of usability, expressiveness, and throughput for real-world use cases?

Key findings

• BeepBeep successfully handles use cases that are difficult or impossible to model in commercial CEP tools or RDBMS, particularly those involving complex event correlations and temporal logic.
• For standard window and pattern queries, BeepBeep achieves a throughput that is lower than top-tier commercial systems but remains reasonable for light- to medium-duty workloads.
• The eSQL language enables relational transparency, allowing standard SQL-like operations to be expressed in a familiar and consistent syntax.
• User-defined processors and grammar extensions via palettes allow for significant customization and domain-specific expressiveness without altering the core engine.
• The modular architecture enables clean, readable, and composable processing chains, making it suitable for tasks ranging from network intrusion detection to software testing.
• Despite not prioritizing peak performance, BeepBeep demonstrates that a simple, extensible system can support complex, real-world stream processing scenarios effectively.