[Paper Review] Study on the Data Processing of the IOT Sensor Network Based on Hadoop Cloud Platform and TWLGA Scheduling Algorithm

Guoyu Li, Kang Yang|arXiv (Cornell University)|Dec 1, 2021

E-commerce and Technology Innovations4 citations

TL;DR

This paper proposes a Hadoop cloud platform integrated with a Time and Workload Genetic Algorithm (TWLGA) for efficient big data processing in IoT sensor networks. By optimizing task scheduling across nodes, the platform enhances data processing efficiency and system compatibility, with experiments demonstrating improved performance in handling massive IoT-generated data streams.

ABSTRACT

The Internet of Things (IOT) sensor network is an effective solution for monitoring environment condition. IOT sensor network generates massive data, and the abilities of massive data storage, processing and query become technical challenges. To solve the problem, a Hadoop cloud platform is proposed. With the help of time and workload genetic algorithm (TWLGA), the data processing platform provides the work of one node to share with others, which not only raises efficiency of one single node, but also provides the compatibility support to reduce the possible risk of software and hardware. In the experiment, a Hadoop cluster platform with TWLGA scheduling algorithm is built, and the performance of the platform is tested. The results show that the Hadoop cloud platform is suitable for big data processing of the IOT sensor network.

Motivation & Objective

• To address the challenge of storing, processing, and querying massive data generated by IoT sensor networks.
• To improve the efficiency and reliability of data processing in distributed IoT environments.
• To reduce hardware and software failure risks through workload and time-aware task scheduling.
• To design and evaluate a Hadoop-based cloud platform enhanced with a novel TWLGA scheduling algorithm.

Proposed method

• A Hadoop cloud platform is implemented to manage large-scale data from IoT sensor networks.
• The Time and Workload Genetic Algorithm (TWLGA) is developed to optimize task scheduling across cluster nodes.
• TWLGA dynamically balances workloads by considering execution time and system resource availability.
• The algorithm uses genetic operations (selection, crossover, mutation) to evolve optimal scheduling solutions.
• The platform enables inter-node resource sharing to improve processing efficiency and fault tolerance.
• A Hadoop cluster with the TWLGA scheduler is deployed and tested using real-world IoT data workloads.

Experimental results

Research questions

• RQ1 How can a cloud-based platform efficiently manage the high-volume data generated by IoT sensor networks?
• RQ2 To what extent does the TWLGA scheduling algorithm improve task distribution and processing speed in a Hadoop cluster?
• RQ3 Can the integration of TWLGA reduce hardware and software failure risks in IoT data processing systems?
• RQ4 What performance gains does the proposed platform achieve compared to traditional scheduling methods?

Key findings

• The Hadoop cloud platform with TWLGA scheduling significantly improves data processing efficiency in IoT sensor networks.
• The TWLGA algorithm reduces task completion time by optimizing workload distribution across nodes.
• Resource sharing among nodes enhances system fault tolerance and reduces dependency on individual node performance.
• The platform demonstrates strong scalability and compatibility in handling large-scale IoT data streams.
• Experimental results confirm the platform's suitability for real-time and high-throughput IoT data processing applications.
• The integration of genetic algorithm-based scheduling leads to more balanced and efficient cluster utilization.