[论文解读] A Multi-Dimensional Big Data Storing System for Generated COVID-19 Large-Scale Data using Apache Spark

Manar A. Elmeiligy, Ali I. El-Desouky+1 more|arXiv (Cornell University)|Apr 30, 2020

Anomaly Detection Techniques and Applications参考文献 15被引用 6

一句话总结

本文提出CSS-COVID，一种基于Apache Spark构建的多维大数据存储系统，可高效管理并查询大规模实时COVID-19数据。通过将数据划分为带索引的子集，并利用Spark在分布式节点上的内存处理能力，该系统显著降低了查询与存储处理时间，在真实数据集上表现出高效率。

ABSTRACT

The ongoing outbreak of coronavirus disease (COVID-19) had burst out in Wuhan China, specifically in December 2019. COVID-19 has caused by a new virus that had not been identified in human previously. This was followed by a widespread and rapid spread of this epidemic throughout the world. Daily, the number of the confirmed cases are increasing rapidly, number of the suspect increases, based on the symptoms that accompany this disease, and unfortunately number of the deaths also increase. Therefore, with these increases in number of cases around the world, it becomes hard to manage all these cases information with different situations; if the patient either injured or suspect with which symptoms that appeared on the patient. Therefore, there is a critical need to construct a multi-dimensional system to store and analyze the generated large-scale data. In this paper, a Comprehensive Storing System for COVID-19 data using Apache Spark (CSS-COVID) is proposed, to handle and manage the problem caused by increasing the number of COVID-19 daily. CSS-COVID helps in decreasing the processing time for querying and storing COVID-19 daily data. CSS-COVID consists of three stages, namely, inserting and indexing, storing, and querying stage. In the inserting stage, data is divided into subsets and then index each subset separately. The storing stage uses set of storing-nodes to store data, while querying stage is responsible for handling the querying processes. Using Apache Spark in CSS-COVID leverages the performance of dealing with large-scale data of the coronavirus disease injured whom increase daily. A set of experiments are applied, using real COVID-19 Datasets, to prove the efficiency of CSS-COVID in indexing large-scale data.

研究动机与目标

• 为应对来自全球病例、症状和死亡人数的快速增长且异构的COVID-19数据管理挑战。
• 降低大规模、每日更新的COVID-19数据集在存储和查询过程中的处理时间。
• 设计一种可扩展的分布式系统，支持多维健康数据的高效索引、存储与检索。
• 利用Apache Spark的内存计算能力，在大数据环境中实现高性能数据处理。

提出的方法

• 系统在插入和索引阶段将传入的COVID-19数据划分为子集。
• 每个数据子集被分配一个唯一索引，以实现快速检索和并行处理。
• 在存储阶段，利用Apache Spark的弹性分布式数据集（RDD）将数据分发到存储节点集群，实现容错能力。
• 在查询阶段，使用Spark的分布式查询执行引擎对已索引数据执行低延迟分析操作。
• Apache Spark的内存计算模型通过最小化I/O开销，加速了数据处理。
• 系统实现为一个三阶段流水线：插入/索引、存储和查询，由Spark的DAG调度器协调执行。

实验结果

研究问题

• RQ1 如何设计一种可扩展的大数据系统，以处理多维COVID-19数据的快速摄入与查询？
• RQ2 在疫情期间，大规模健康数据管理中，哪些架构组件是减少处理延迟所必需的？
• RQ3 Apache Spark在实时、高速度的健康数据中，能在多大程度上提升索引与查询性能？
• RQ4 与传统关系型数据库相比，所提出的系统在处理大规模、动态数据集时，其处理时间优势如何？

主要发现

• CSS-COVID系统通过利用Apache Spark的内存处理能力，显著减少了查询与存储处理时间。
• 独立索引各数据子集可实现高效的并行处理与更快的数据检索。
• 通过在多个节点上分布存储，系统展现出高可扩展性与容错能力。
• 使用真实COVID-19数据集的实验评估证实，该系统能高效处理大规模、动态的数据工作负载。
• 三阶段架构——插入/索引、存储和查询——在管理异构、时间敏感的健康数据方面被证明是有效的。
• 使用Spark的RDD和DAG执行模型，即使在高数据摄入速率下，也能确保低延迟性能。