[Paper Review] RT-DAP: A Real-Time Data Analytics Platform for Large-scale Industrial Process Monitoring and Control

Song Han, Tao Gong+4 more|arXiv (Cornell University)|Feb 21, 2018

Time Series Analysis and Forecasting4 citations

TL;DR

RT-DAP is a real-time data analytics platform designed for large-scale industrial process monitoring and control, leveraging a distributed time-series database, unified messaging protocol, and real-time processing framework on Microsoft Azure. It enables low-latency data streaming, aggregation, and visualization, reducing HBase I/O overhead through adaptive batching and achieving sub-500s processing for 1000 tags/sec with optimal queue sizing.

ABSTRACT

In most process control systems nowadays, process measurements are periodically collected and archived in historians. Analytics applications process the data, and provide results offline or in a time period that is considerably slow in comparison to the performance of the manufacturing process. Along with the proliferation of Internet-of-Things (IoT) and the introduction of "pervasive sensors" technology in process industries, increasing number of sensors and actuators are installed in process plants for pervasive sensing and control, and the volume of produced process data is growing exponentially. To digest these data and meet the ever-growing requirements to increase production efficiency and improve product quality, there needs to be a way to both improve the performance of the analytics system and scale the system to closely monitor a much larger set of plant resources. In this paper, we present a real-time data analytics platform, called RT-DAP, to support large-scale continuous data analytics in process industries. RT-DAP is designed to be able to stream, store, process and visualize a large volume of realtime data flows collected from heterogeneous plant resources, and feedback to the control system and operators in a realtime manner. A prototype of the platform is implemented on Microsoft Azure. Our extensive experiments validate the design methodologies of RT-DAP and demonstrate its efficiency in both component and system levels.

Motivation & Objective

• Address the growing gap between real-time industrial data generation and slow, offline analytics in process control systems.
• Enable large-scale, real-time monitoring and control of heterogeneous plant resources using pervasive sensor data.
• Overcome limitations of traditional historians and general-purpose platforms in handling high-velocity, time-series industrial data.
• Design a scalable, low-latency analytics stack integrating data ingestion, storage, processing, and visualization for industrial applications.
• Close the feedback loop between real-time analytics and control systems to improve process efficiency, safety, and product quality.

Proposed method

• Design a unified messaging protocol to standardize real-time data ingestion from diverse plant resources.
• Implement a distributed time-series database (HBase-based) optimized for high-throughput, low-latency time-series data storage and querying.
• Develop an analytics model development studio (MDS) for designing data and control flow pipelines for process analytics.
• Integrate an elastic real-time processing engine (Storm) with adaptive batching to reduce HBase I/O operations and latency.
• Deploy a lightweight IIoT field gateway (IIoT-FG) on Minnowboard to connect heterogeneous sensors and actuators to the platform.
• Use a dual-threaded Spout architecture with self-adaptive queueing to balance throughput and latency in data aggregation.

Experimental results

Research questions

• RQ1 How can a real-time analytics platform be architected to handle high-velocity, heterogeneous time-series data from large-scale industrial plants?
• RQ2 What design patterns minimize I/O overhead in time-series databases during continuous data aggregation?
• RQ3 How can batching and adaptive queueing improve the performance of real-time stream processing in industrial analytics workloads?
• RQ4 To what extent can a cloud-based platform like Microsoft Azure support low-latency, scalable analytics for industrial process control?
• RQ5 Can the integration of a field gateway and real-time processing framework close the loop between analytics and control decisions effectively?

Key findings

• The RT-DAP platform achieved sub-500s end-to-end processing time for data streams up to 1000 tags per second when using optimal queue sizing.
• HBase write and scan operations dominated processing time, accounting for the majority of latency in data aggregation tasks.
• Increasing the maximum queue size from 20 to 200 reduced HBase write time significantly, improving system throughput and reducing latency.
• When the queue size exceeded 160, the average runtime queue size stabilized below the maximum, indicating effective load balancing.
• The system could not keep up with data rates above 1200 tags per second, as processing time exceeded the 500-second test window.
• Adaptive batching reduced HBase access frequency by processing multiple records in a single batch, improving I/O efficiency without sacrificing real-time responsiveness.