[论文解读] Why Atomicity Matters to AI/ML Infrastructure: Snapshots, Firmware Updates, and the Cost of the Forward-In-Time-Only Category Mistake

Paul Borrill|arXiv (Cornell University)|Mar 3, 2026

Distributed systems and fault tolerance被引用 0

一句话总结

论文认为在异步崩溃恢复设置下无法保证原子检查点和原子固件部署，并且将检查点/升级事件视为时间边界是一种范式错误；提出基于收敛性的替代方案。

ABSTRACT

Large-scale AI/ML training systems depend on two assumptions that are rarely examined: (1) that checkpoints represent atomic snapshots of global training state, and (2) that infrastructure updates can be applied without inducing mixed-protocol cluster states. Both assumptions are instances of a deeper structural error: the Forward-In-Time-Only (FITO) category mistake, which confuses protocol convergence properties with temporal predicates. We formalize this confusion as a type error: the identification of a temporal snapshot $\mathsf{Snap}(t)$ with a convergence property $\mathsf{Conv}(\mathcal{P},e)$. We model checkpoint execution in a process-algebraic framework and prove that under asynchronous composition with crash-recovery failures, no temporal instant can serve as an atomicity boundary. We reformulate checkpoint inconsistency on an epoch lattice and show that atomicity is a measure-zero event whose complement grows exponentially with the number of independent persistence domains. We formalize mixed-epoch recovery as a type violation in the optimization algebra and show that the resulting update is not a valid step of any standard optimizer. For firmware fleet updates, we strengthen the known consensus-hardness result: atomic deployment requires not merely agreement but common knowledge of the epoch transition, which is strictly unattainable in asynchronous systems with unreliable communication. We conclude by sketching a bilateral convergence protocol, inspired by Open Atomic Ethernet, that achieves $\mathsf{Conv}(\mathcal{P},e)$ without requiring $\mathsf{Snap}(t)$ -- replacing the FITO assumption with constraint semantics.

研究动机与目标

• Formalize the Forward-In-Time-Only (FITO) category mistake in AI/ML infrastructure.
• Model checkpointing and firmware updates as asynchronous process compositions to distinguish trace properties from time predicates.
• Prove the non-existence of temporal snapshot boundaries under crash-recovery failures.
• Reframe checkpoint consistency on an epoch lattice and quantify atomicity as a measure-zero event.
• Propose convergence-based protocols as alternatives to temporal boundaries for checkpoints and upgrades.

提出的方法

• Formal process-algebraic modeling of checkpointing as asynchronous composition of persistence processes.
• Define Snap(t,e) as a temporal snapshot predicate and Conv(P,e) as a protocol convergence property; prove they are distinct types.
• Use an asynchronous crash-recovery failure model with independent failure domains to derive impossibility results for temporal boundaries.
• Introduce epoch lattices and measure-theoretic arguments to show atomicity is measure-zero as persistence domains grow.

实验结果

研究问题

• RQ1 Can a temporal boundary t_c exist that guarantees atomic commitment across all components under asynchronous crash-recovery?
• RQ2 Is atomic deployment of firmware achievable under asynchrony without common knowledge of the epoch transition?
• RQ3 How does mixed-epoch recovery affect standard optimizer updates in AI/ML training?
• RQ4 What convergence-based mechanisms can replace temporal snapshots to ensure consistent global state?
• RQ5 What practical protocols can approximate atomicity given the impossibility results?

主要发现

• Checkpoint atomicity is a measure-zero event in large-scale systems with many persistence units.
• Under independent crash-recovery failures, no asynchronous checkpoint protocol can guarantee a temporal boundary where all components reflect the same committed epoch.
• Mixed-epoch recovery causes optimizer steps (e.g., AdamW) to become invalid with respect to any single epoch trajectory.
• Common knowledge of the epoch transition is unattainable in asynchronous systems, making atomic firmware deployment impossible with purely message-based coordination.
• A bilateral convergence protocol can achieve Conv(P,e) without relying on a temporal snapshot boundary, reframing checkpointing from time-based to protocol-based convergence.