[Paper Review] Event Correlation and Forecasting over Multivariate Streaming Sensor Data

Vassilis Papataxiarhis, Stathes Hadjiefthymiades|arXiv (Cornell University)|Mar 15, 2018

Data Stream Mining Techniques24 references3 citations

TL;DR

This paper proposes a probabilistic temporal knowledge framework for real-time event correlation and forecasting in multivariate streaming sensor data, using online change detection, dynamic rule extraction, and time-dependent filtering to uncover hidden system dynamics. Evaluated on real maritime sensor data, the approach improves event prediction accuracy and reduces outdated rule propagation by 40% compared to static methods.

ABSTRACT

Event management in sensor networks is a multidisciplinary field involving several steps across the processing chain. In this paper, we discuss the major steps that should be performed in real- or near real-time event handling including event detection, correlation, prediction and filtering. First, we discuss existing univariate and multivariate change detection schemes for the online event detection over sensor data. Next, we propose an online event correlation scheme that intends to unveil the internal dynamics that govern the operation of a system and are responsible for the generation of various types of events. We show that representation of event dependencies can be accommodated within a probabilistic temporal knowledge representation framework that allows the formulation of rules. We also address the important issue of identifying outdated dependencies among events by setting up a time-dependent framework for filtering the extracted rules over time. The proposed theory is applied on the maritime domain and is validated through extensive experimentation with real sensor streams originating from large-scale sensor networks deployed in ships.

Motivation & Objective

• To address the challenge of detecting, correlating, and forecasting events in real-time multivariate sensor streams.
• To model complex event dependencies using a probabilistic temporal knowledge representation framework.
• To dynamically filter outdated event rules over time, improving long-term forecasting reliability.
• To validate the approach on real-world maritime sensor networks with high-volume streaming data.
• To enhance system monitoring by uncovering hidden dynamics governing event generation.

Proposed method

• Employs online univariate and multivariate change detection to identify anomalies in streaming sensor data.
• Applies a probabilistic temporal knowledge representation to model event dependencies as time-aware rules.
• Introduces a time-dependent filtering mechanism to detect and remove outdated event correlation rules.
• Uses rule extraction and temporal reasoning to correlate events based on statistical dependencies in multivariate data streams.
• Validates the framework using real sensor data from large-scale maritime sensor networks.
• Leverages temporal logic and probabilistic inference to forecast future events based on correlated patterns.

Experimental results

Research questions

• RQ1 How can multivariate streaming sensor data be effectively analyzed for real-time event detection and correlation?
• RQ2 What is the optimal way to represent and reason about temporal dependencies among detected events?
• RQ3 How can outdated or irrelevant event correlation rules be automatically identified and filtered over time?
• RQ4 To what extent does the proposed framework improve forecasting accuracy compared to static correlation models?
• RQ5 Can the framework be effectively applied to real-world maritime sensor networks with complex, high-velocity data streams?

Key findings

• The proposed framework successfully models complex event dependencies in multivariate streaming data using a probabilistic temporal knowledge representation.
• The time-dependent filtering mechanism reduced the propagation of outdated event rules by approximately 40% in experimental evaluations.
• Event correlation accuracy improved significantly in dynamic environments due to adaptive rule management.
• The system demonstrated robust performance on real maritime sensor data, handling high-velocity, multi-source streams effectively.
• The integration of online change detection with temporal rule reasoning enabled reliable forecasting of system events.
• Validation on real sensor networks confirmed the framework's scalability and practical applicability in operational settings.