[Paper Review] Silent Data Corruptions at Scale

Harish Dattatraya Dixit, Sneha Pendharkar+5 more|arXiv (Cornell University)|Feb 22, 2021

Radiation Effects in ElectronicsEngineering21 references68 citations

TL;DR

The paper analyzes silent data corruptions (SDCs) in datacenter CPUs, presents a real-world debug case, and discusses detection and mitigation strategies at scale.

ABSTRACT

Silent Data Corruption (SDC) can have negative impact on large-scale infrastructure services. SDCs are not captured by error reporting mechanisms within a Central Processing Unit (CPU) and hence are not traceable at the hardware level. However, the data corruptions propagate across the stack and manifest as application-level problems. These types of errors can result in data loss and can require months of debug engineering time. In this paper, we describe common defect types observed in silicon manufacturing that leads to SDCs. We discuss a real-world example of silent data corruption within a datacenter application. We provide the debug flow followed to root-cause and triage faulty instructions within a CPU using a case study, as an illustration on how to debug this class of errors. We provide a high-level overview of the mitigations to reduce the risk of silent data corruptions within a large production fleet. In our large-scale infrastructure, we have run a vast library of silent error test scenarios across hundreds of thousands of machines in our fleet. This has resulted in hundreds of CPUs detected for these errors, showing that SDCs are a systemic issue across generations. We have monitored SDCs for a period longer than 18 months. Based on this experience, we determine that reducing silent data corruptions requires not only hardware resiliency and production detection mechanisms, but also robust fault-tolerant software architectures.

Motivation & Objective

• Identify defect types in silicon manufacturing that lead to SDCs.
• Demonstrate how SDCs propagate to application level using a real-world case study.
• Describe debugging workflows and tools to root-cause SDCs at scale across fleets.
• Outline hardware and software strategies to reduce SDC risk in production environments.

Proposed method

• Classify silicon defect categories: device errors, early life failures, degradation, and end-of-life wear-out.
• Analyze a real-world Spark-based application showing SDC propagation to missing data and possible data loss.
• Detail a multi-language reproducer workflow from Scala to Java Byte Code to assembly for root-cause analysis.
• Propose best-practice guidelines for assembling deterministic reproducers and debugging at instruction level.

Experimental results

Research questions

• RQ1 What silicon and manufacturing defect categories contribute to silent data corruptions in datacenter CPUs?
• RQ2 How do SDCs propagate from hardware through software stacks to cause application-level failures?
• RQ3 What debugging workflows and tooling enable root-cause analysis of SDCs at scale?
• RQ4 What detection and fault-tolerant software/hardware strategies can mitigate SDCs in large fleets?

Key findings

• SDCs in datacenter CPUs occur at higher rates than traditional soft-error FIT models and are repeatable at scale.
• A real-world case shows SDCs can cause missing or corrupted data in decompression and data processing workflows.
• Debugging at scale requires cross-language reproducers and assembly-level tracing to identify faulty instructions.
• Mitigations include hardware protections, targeted testing, detection mechanisms, and software fault-tolerant designs.