[Paper Review] An Automated Data Engineering Pipeline for Anomaly Detection of IoT Sensor Data

Xinze Li, Baixi Zou|arXiv (Cornell University)|Sep 28, 2021

Anomaly Detection Techniques and Applications24 references4 citations

TL;DR

This paper proposes an automated, end-to-end data engineering pipeline for real-time anomaly detection in IoT sensor data using a hybrid cloud-edge architecture. Leveraging Raspberry Pi, AWS services (IoT Core, S3, DynamoDB, Lambda, Greengrass), and machine learning models (Isolation Forest, K-means), the system detects anomalous patterns such as garage door openings during indoor activity, enabling a proactive anti-theft smart home security system with 55,998 anomalies detected in garage data and 12,700 in bedroom data.

ABSTRACT

The rapid development in the field of System of Chip (SoC) technology, Internet of Things (IoT), cloud computing, and artificial intelligence has brought more possibilities of improving and solving the current problems. With data analytics and the use of machine learning/deep learning, it is made possible to learn the underlying patterns and make decisions based on what was learned from massive data generated from IoT sensors. When combined with cloud computing, the whole pipeline can be automated, and free of manual controls and operations. In this paper, an implementation of an automated data engineering pipeline for anomaly detection of IoT sensor data is studied and proposed. The process involves the use of IoT sensors, Raspberry Pis, Amazon Web Services (AWS) and multiple machine learning techniques with the intent to identify anomalous cases for the smart home security system.

Motivation & Objective

• To automate the end-to-end data pipeline for anomaly detection in real-time IoT sensor data, reducing manual intervention.
• To design a scalable, cloud-integrated architecture that supports edge data collection, cloud processing, and model deployment.
• To evaluate and deploy multiple machine learning models (Isolation Forest, K-means) for identifying anomalous behavior in smart home environments.
• To detect specific intrusion scenarios—such as garage door opening while occupants are active indoors—using sensor pattern analysis.
• To demonstrate the feasibility of using automated data pipelines for real-world security applications in residential IoT systems.

Proposed method

• Utilizes Raspberry Pi devices with embedded sensors (accelerometer, gyroscope, magnetometer, light, temperature, humidity) to collect real-time IoT data at the edge.
• Employs AWS IoT Core for secure, scalable device-to-cloud communication using MQTT and LoRaWAN protocols.
• Stores raw and processed sensor data in AWS DynamoDB (for real-time access) and S3 (for long-term archival) with lifecycle policies.
• Applies AWS IoT Analytics to preprocess and enrich time-series sensor data before model training.
• Deploys serverless AWS Lambda functions via AWS Greengrass to execute data ingestion and model inference directly on Raspberry Pi edge devices.
• Uses Isolation Forest and K-means clustering to detect anomalies, followed by cluster analysis to interpret anomalous behavior patterns.

Experimental results

Research questions

• RQ1 How can an automated, end-to-end data engineering pipeline be designed to support real-time anomaly detection in distributed IoT sensor networks?
• RQ2 What combination of cloud services and edge computing components enables scalable, low-latency, and maintainable IoT data processing for anomaly detection?
• RQ3 To what extent can Isolation Forest and K-means clustering effectively identify meaningful anomalous patterns in real-world smart home sensor data?
• RQ4 Can distinct anomaly clusters be semantically interpreted as specific real-world events (e.g., garage door opening during indoor activity)?
• RQ5 How do feature-level differences in temperature, lux, and humidity across clusters help identify abnormal but non-threatening behaviors (e.g., working in a bedroom with lights on)?

Key findings

• The pipeline successfully detected 55,998 anomalies out of 1,119,942 data points in the garage environment using Isolation Forest, with anomalies concentrated in high-lux and high-temperature conditions.
• K-means clustering of anomalies in the garage data revealed four distinct clusters, with cluster 0 showing elevated temperature, humidity, and lux levels—indicative of garage door opening and air exchange.
• In the bedroom environment, 12,700 anomalies were detected, with cluster 2 showing significantly higher mean temperature and lux values, consistent with human presence and device usage (e.g., computers, lights).
• The combination of elevated temperature and lux in the garage during low-lux normal conditions was interpreted as a strong indicator of potential intrusion, such as an open garage door during indoor activity.
• Feature analysis confirmed that abnormal clusters were not random but reflected meaningful real-world behaviors: e.g., high temperature and light levels in the bedroom indicated active human presence, while elevated garage conditions suggested environmental changes due to door opening.
• The system demonstrated that automated pipeline integration with AWS services enables real-time, interpretable anomaly detection suitable for smart home security applications.