[Paper Review] From Kepler to Newton: Inductive Biases Guide Learned World Models in Transformers

Ziming Liu, Sophia Sanborn+2 more|arXiv (Cornell University)|Feb 6, 2026

Machine Learning in Materials Science0 citations

TL;DR

The paper shows that three simple inductive biases (spatial smoothness, spatial stability, and temporal locality) enable transformers to learn true physical world models, revealing Newtonian forces or Keplerian orbit representations depending on context length.

ABSTRACT

Can general-purpose AI architectures go beyond prediction to discover the physical laws governing the universe? True intelligence relies on "world models" -- causal abstractions that allow an agent to not only predict future states but understand the underlying governing dynamics. While previous "AI Physicist" approaches have successfully recovered such laws, they typically rely on strong, domain-specific priors that effectively "bake in" the physics. Conversely, Vafa et al. recently showed that generic Transformers fail to acquire these world models, achieving high predictive accuracy without capturing the underlying physical laws. We bridge this gap by systematically introducing three minimal inductive biases. We show that ensuring spatial smoothness (by formulating prediction as continuous regression) and stability (by training with noisy contexts to mitigate error accumulation) enables generic Transformers to surpass prior failures and learn a coherent Keplerian world model, successfully fitting ellipses to planetary trajectories. However, true physical insight requires a third bias: temporal locality. By restricting the attention window to the immediate past -- imposing the simple assumption that future states depend only on the local state rather than a complex history -- we force the model to abandon curve-fitting and discover Newtonian force representations. Our results demonstrate that simple architectural choices determine whether an AI becomes a curve-fitter or a physicist, marking a critical step toward automated scientific discovery.

Motivation & Objective

• Motivate the search for internal world models in foundation models beyond prediction.
• Identify minimal inductive biases that enable learning physical laws from data.
• Systematically contrast regression vs. classification formulations for continuous dynamics.
• Demonstrate how context length controls the emergence of Newtonian vs. Keplerian world models.
• Provide guidance toward mechanistic understanding and out-of-distribution generalization in AI scientists.

Proposed method

• Identify three inductive biases: spatial smoothness, spatial stability, and temporal locality.
• Compare tokenization-based classification with continuous regression on Kepler-like data and controlled Kepler datasets.
• Use linear probes to test whether internal representations encode spatial coordinates or force-related quantities.
• Vary vocabulary size, training data, embedding dimension, and context length to study emergent world models.
• Demonstrate that regression with noisy contexts reduces error accumulation and outperforms classification under optimized hyperparameters.
• Expose a phase transition: short context yields Newtonian (force-based) models, long context yields Keplerian (ellipse-based) models.

Figure 1: Visual abstract. Top left: The problem setup of Vafa et al. ( 2025 ) : planetary motion prediction is formulated as next token(s) prediction. Bottom left: Inductive biases are key to learning Newtonian world models. Three inductive biases are identified and used to fix respective failure m

Experimental results

Research questions

• RQ1 Why do transformers fail to learn Newtonian world models for planetary motion, and what minimal biases can fix this?
• RQ2 Can simple inductive biases enable emergence of mechanistic (Newtonian) versus geometric (Keplerian) world models in transformers?
• RQ3 How do data size, vocabulary size, and context length affect the learned world model and predictive robustness?
• RQ4 What are the comparative benefits of regression versus classification formulations for learning continuous dynamics?

Key findings

• Three biases are enough to steer world-model learning in transformers: spatial smoothness improves spatial mapping, spatial stability mitigates error accumulation in regression via noisy contexts, and temporal locality shifts the model toward Newtonian dynamics.
• Regression with continuous coordinates, augmented by noisy contexts, outperforms discrete classification across data scales, given optimized hyperparameters (context noise sigma, vocabulary size V).
• Context length governs the emergent world model: small context (2) yields Newtonian force-based representations; large context yields Keplerian ellipses with parameters a, b, and Laplace–Runge–Runge vector encoding nearly perfectly (R^2 ≈ 0.998).
• Keplerian features (ellipse parameters) are captured with high linear predictability when context is long (R^2 ≈ 0.998 for a, b, A) whereas Newtonian force variables are linearly encoded with short context (R^2 ≈ 0.999).
• Tokenization with a very large vocabulary can hinder spatial map emergence; reducing vocabulary size or using regression improves spatial representation under comparable compute.

Figure 2: Analyzing the embeddings of the transformer model used in Vafa et al. ( 2025 ) . (a) Illustration of training dynamics of token embeddings: embeddings are randomly initialized (left), gradually gain spatial structure during training (middle), requiring substantial compute and data to reach