[Paper Review] Strider: A Hybrid Adaptive Distributed RDF Stream Processing Engine

Xiangnan Ren, Olivier Curé|arXiv (Cornell University)|May 16, 2017

Advanced Database Systems and Queries19 references4 citations

TL;DR

Strider is a hybrid adaptive distributed RDF stream processing engine built on Apache Spark and Kafka that dynamically optimizes SPARQL query plans at runtime based on data stream characteristics. It achieves up to 3.1 million RDF triples per second throughput on a 9-node cluster, demonstrating a 60x performance gain over state-of-the-art systems in single-machine evaluation.

ABSTRACT

Real-time processing of data streams emanating from sensors is becoming a common task in Internet of Things scenarios. The key implementation goal consists in efficiently handling massive incoming data streams and supporting advanced data analytics services like anomaly detection. In an on-going, industrial project, we found out that a 24/7 available stream processing engine usually faces dynamically changing data and workload characteristics. These changes impact the engine's performance and reliability. We propose Strider, a hybrid adaptive distributed RDF Stream Processing engine that optimizes logical query plan according to the state of data streams. Strider has been designed to guarantee important industrial properties such as scalability, high availability, fault-tolerant, high throughput and acceptable latency. These guarantees are obtained by designing the engine's architecture with state-of-the-art Apache components such as Spark and Kafka. We highlight the efficiency (e.g., on a single machine machine, up to 60x gain on throughput compared to state-of-the-art systems, a throughput of 3.1 million triples/second on a 9 machines cluster, a major breakthrough in this system's category) of Strider on real-world and synthetic data sets.

Motivation & Objective

• Address the challenge of maintaining high throughput and low latency in real-time RDF stream processing under dynamically changing data and workload conditions.
• Design a production-ready RSP engine that supports scalability, high availability, fault tolerance, and adaptivity for industrial IoT workloads.
• Overcome the limitations of static, compile-time query plans in existing distributed RSP systems by introducing runtime adaptation based on stream volume and query characteristics.
• Enable efficient continuous SPARQL processing over large-scale, schema-free RDF data streams from sensor networks in real-world deployments.
• Bridge the gap in the RSP ecosystem by supporting micro-batch processing with adaptive optimization strategies for improved performance and reliability.

Proposed method

• Leverages Apache Kafka for high-throughput, fault-tolerant message streaming and data ingestion from distributed sensors.
• Employs Spark Streaming with micro-batch processing to enable near real-time evaluation of continuous SPARQL queries over time-bounded data windows.
• Introduces a hybrid adaptive query processing (AQP) strategy that switches between rule-based (compile-time) and cost-based (run-time) optimization based on incoming data volume.
• Dynamically re-optimizes query plans either at the end of the previous window or during the current window to adapt to changing stream characteristics.
• Uses a trigger-based mechanism in the optimizer to enable automatic strategy switching during query execution without manual intervention.
• Applies logical plan restructuring for complex queries involving multiple joins, especially to mitigate performance degradation in structurally unstable RDF streams.

Experimental results

Research questions

• RQ1 How can a distributed RDF stream processing engine maintain high throughput and low latency under dynamically changing data stream workloads?
• RQ2 What mechanisms enable effective adaptation of query plans in real-time RDF stream processing systems without sacrificing performance or reliability?
• RQ3 To what extent can hybrid adaptive query processing strategies—combining rule-based and cost-based optimization—improve performance in large-scale RDF stream processing?
• RQ4 How does micro-batch processing with runtime plan adaptation compare to record-at-a-time processing in terms of throughput and latency for continuous SPARQL queries?
• RQ5 What are the performance bottlenecks in existing distributed RSP systems, and how can they be mitigated through architectural choices like Spark and Kafka integration?

Key findings

• Strider achieves a throughput of 3.1 million RDF triples per second on a 9-machine cluster, representing a major performance breakthrough in the distributed RSP domain.
• On a single machine, Strider demonstrates up to a 60x improvement in throughput compared to state-of-the-art systems, highlighting its efficiency in resource-constrained environments.
• The adaptive query optimization strategy effectively handles structurally unstable RDF streams, such as those with complex join patterns in query Q9, preventing sharp performance degradation.
• The system exhibits strong reliability in processing real-world and synthetic data sets, maintaining consistent performance across varying stream characteristics.
• The micro-batch architecture enables better fault tolerance and system adaptivity compared to record-at-a-time systems, despite a slight increase in latency.
• Limitations include a 0.8–1 second initialization overhead for data preparation and reduced throughput with increasing numbers of join operators, suggesting potential for optimization via cluster scaling or enhanced driver configuration.