[论文解读] From Kepler to Newton: Inductive Biases Guide Learned World Models in Transformers

Ziming Liu, Sophia Sanborn+2 more|arXiv (Cornell University)|Feb 6, 2026

Machine Learning in Materials Science被引用 0

一句话总结

论文表明三种简单的归纳偏置（空间平滑、空间稳定性和时序局部性）使变换模型能够学习真实的物理世界模型，根据上下文长度呈现牛顿力学或开普勒轨道表示。

ABSTRACT

Can general-purpose AI architectures go beyond prediction to discover the physical laws governing the universe? True intelligence relies on "world models" -- causal abstractions that allow an agent to not only predict future states but understand the underlying governing dynamics. While previous "AI Physicist" approaches have successfully recovered such laws, they typically rely on strong, domain-specific priors that effectively "bake in" the physics. Conversely, Vafa et al. recently showed that generic Transformers fail to acquire these world models, achieving high predictive accuracy without capturing the underlying physical laws. We bridge this gap by systematically introducing three minimal inductive biases. We show that ensuring spatial smoothness (by formulating prediction as continuous regression) and stability (by training with noisy contexts to mitigate error accumulation) enables generic Transformers to surpass prior failures and learn a coherent Keplerian world model, successfully fitting ellipses to planetary trajectories. However, true physical insight requires a third bias: temporal locality. By restricting the attention window to the immediate past -- imposing the simple assumption that future states depend only on the local state rather than a complex history -- we force the model to abandon curve-fitting and discover Newtonian force representations. Our results demonstrate that simple architectural choices determine whether an AI becomes a curve-fitter or a physicist, marking a critical step toward automated scientific discovery.

研究动机与目标

• 在基础模型中寻找除了预测之外的内部世界模型的动机。
• 识别使从数据学习物理定律所需的最小归纳偏置。
• 系统性对比回归与分类在连续动力学中的公式化。
• 演示上下文长度如何控制牛顿ian与开普勒ian世界模型的出现。
• 为AI科学家在机制理解与分布外泛化方面提供指导。

提出的方法

• 识别三种归纳偏置：空间平滑性、空间稳定性和时序局部性。
• 在开普勒样数据及受控开普勒数据集上，将基于标记的分类与连续回归进行比较。
• 使用线性探针测试内部表征是否编码空间坐标或与力相关的量。
• 改变词汇量、训练数据、嵌入维度和上下文长度以研究涌现的世界模型。
• 证明在带噪声上下文的回归在优化超参数下减少误差累积并优于分类。
• 揭示相变：短上下文产生牛顿力学（基于力）的模型，长上下文产生开普勒（椭圆）模型。

Figure 1: Visual abstract. Top left: The problem setup of Vafa et al. ( 2025 ) : planetary motion prediction is formulated as next token(s) prediction. Bottom left: Inductive biases are key to learning Newtonian world models. Three inductive biases are identified and used to fix respective failure m

实验结果

研究问题

• RQ1 为什么变换模型难以学习行星运动的牛顿世界模型，哪些最小偏置可以修复？
• RQ2 简单的归纳偏置能否使变换模型涌现出机械性（牛顿）与几何性（开普勒）的世界模型？
• RQ3 数据量、词汇量和上下文长度如何影响学习到的世界模型和预测鲁棒性？
• RQ4 回归与分类在学习连续动力学方面各自的优点是什么？

主要发现

• 三种偏置足以引导变换模型的世界模型学习：空间平滑性改善空间映射，空间稳定性通过带噪声上下文的回归减少误差累积，时序局部性将模型倾向于牛顿动力学。
• 在带噪声上下文的连续坐标回归在给定优化超参数（上下文噪声sigma、词汇量V）下在各数据规模上优于离散分类。
• 上下文长度决定涌现的世界模型：较短上下文（2）产生牛顿力基表示；较长上下文产生带参数a、b的开普勒椭圆表示，且拉普拉斯-亨格-鲁恩向量编码几乎完美（R^2 ≈ 0.998）。
• 当上下文较长时，开普勒特征（椭圆参数）具有高线性可预测性（a、b、A的R^2≈0.998）；而牛顿力变量在短上下文中可线性编码（R^2≈0.999）。
• 非常大的词汇量的标记化可能阻碍空间映射的出现；在可比较的计算条件下，减小词汇量或使用回归可改善空间表示。

Figure 2: Analyzing the embeddings of the transformer model used in Vafa et al. ( 2025 ) . (a) Illustration of training dynamics of token embeddings: embeddings are randomly initialized (left), gradually gain spatial structure during training (middle), requiring substantial compute and data to reach