[论文解读] Durability and Availability of Erasure-Coded Storage Systems with Concurrent Maintenance

Şuayb S. Arslan|arXiv (Cornell University)|Jan 22, 2023

Advanced Data Storage Technologies被引用 4

一句话总结

本文提出了一种广义的马尔可夫模型，用于估算具有并发维护的纠删码存储系统中的平均数据丢失时间（MTTDL），并引入了硬错误、扇区故障以及依赖时间的修复等现实故障动态。该研究通过先进的随机模型和仿真框架，提升了可靠性预测的准确性，尤其适用于冷存储和光存储系统。

ABSTRACT

This initial version of this document was written back in 2014 for the sole purpose of providing fundamentals of reliability theory as well as to identify the theoretical types of machinery for the prediction of durability/availability of erasure-coded storage systems. Since the definition of a "system" is too broad, we specifically focus on warm/cold storage systems where the data is stored in a distributed fashion across different storage units with or without continuous operation. The contents of this document are dedicated to a review of fundamentals, a few major improved stochastic models, and several contributions of my work relevant to the field. One of the contributions of this document is the introduction of the most general form of Markov models for the estimation of mean time to failure. This work was partially later published in IEEE Transactions on Reliability. Very good approximations for the closed-form solutions for this general model are also investigated. Various storage configurations under different policies are compared using such advanced models. Later in a subsequent chapter, we have also considered multi-dimensional Markov models to address detached drive-medium combinations such as those found in optical disk and tape storage systems. It is not hard to anticipate such a system structure would most likely be part of future DNA storage libraries. This work is partially published in Elsevier Reliability and System Safety. Topics that include simulation modelings for more accurate estimations are included towards the end of the document by noting the deficiencies of the simplified canonical as well as more complex Markov models, due mainly to the stationary and static nature of Markovinity. Throughout the document, we shall focus on concurrently maintained systems although the discussions will only slightly change for the systems repaired one device at a time.

研究动机与目标

• 开发一种广义的马尔可夫模型，以准确预测在并发维护条件下纠删码存储系统的持久性和可用性。
• 解决经典马尔可夫模型在处理依赖时间的故障率和修复率以及多磁盘容错系统中无记忆假设方面的局限性。
• 通过将潜在扇区错误和不可恢复位错误作为主要故障类型，改进可靠性估计。
• 通过多维马尔可夫模型对脱离的磁盘-介质组合进行精确建模，实现对冷存储系统的有效建模。
• 通过高保真仿真技术（尤其是重要性采样和HFR仿真器）验证并增强可靠性预测。

提出的方法

• 提出一种广义的连续时间马尔可夫链模型，其状态转移表示磁盘故障、修复和数据丢失，从而实现 MTTDL 的闭式估计。
• 引入依赖时间的故障率和修复率函数，以建模非指数分布，克服静态马尔可夫模型的局限性。
• 通过时间依赖的概率函数 $ P_S(t) = (1 - e^{-l(t mod T_S)})P_e $ 实现扇区级故障建模，考虑数据擦除和写入负载的影响。
• 将该模型应用于高级编码方案（如金字塔码和基于MDS的二维阵列），实现不同编码方案之间的性能比较。
• 采用基于仿真的可靠性估计方法，利用重要性采样加速高度容错系统中罕见数据丢失事件的检测。
• 开发一种数据辅助建模框架，整合真实世界故障数据，以提高模型保真度并减少对理想化假设的依赖。

实验结果

研究问题

• RQ1 如何广义化马尔可夫模型，以在具有并发维护和依赖时间故障/修复率的纠删码系统中准确估计 MTTDL？
• RQ2 潜在扇区错误和不可恢复位错误对系统持久性的影响是什么？如何在随机框架内对它们进行建模？
• RQ3 在现实故障动态下，不同编码方案（如复制、MDS、XOR 基编码）在 MTTDL 方面有何比较差异？
• RQ4 多维马尔可夫模型能否有效表示具有脱离磁盘-介质对的冷存储系统（如光存储或磁带库）？
• RQ5 基于仿真的方法（如 HFR 和重要性采样）在估计罕见数据丢失事件方面，相较于分析模型的优越程度如何？

主要发现

• 广义马尔可夫模型在估计 MTTDL 方面比经典模型更准确，尤其是在引入依赖时间的故障和修复动态时。
• 通过 $ P_S(t) $ 引入扇区错误显著提升了故障率估计的准确性，因为潜在扇区错误的发生频率高于硬位错误。
• 该模型表明，MTTDL 对修复率的波动性和故障检测延迟高度敏感，尤其是在重建阶段。
• 使用重要性采样的仿真可显著减少罕见数据丢失事件的计算时间，同时保持低方差。
• HFR 仿真器通过追踪单个磁盘和扇区故障，实现对数据丢失条件的精确检测，其性能优于标准仿真器。
• 研究表明，当真实世界故障模式不具备无记忆特性时，传统马尔可夫模型中的指数假设会低估系统的持久性。