[Paper Review] How Far is Video Generation from World Model: A Physical Law Perspective

Bingyi Kang, Yue Yang+6 more|arXiv (Cornell University)|Nov 4, 2024

Data Visualization and Analytics4 citations

TL;DR

This paper investigates whether video generation models can discover fundamental physical laws from visual data alone, using a controlled 2D simulation environment governed by classical mechanics. Despite strong in-distribution and combinatorial generalization with scaling, models fail on out-of-distribution scenarios due to case-based reasoning prioritizing color, size, velocity, and shape over abstract rule learning.

ABSTRACT

OpenAI's Sora highlights the potential of video generation for developing world models that adhere to fundamental physical laws. However, the ability of video generation models to discover such laws purely from visual data without human priors can be questioned. A world model learning the true law should give predictions robust to nuances and correctly extrapolate on unseen scenarios. In this work, we evaluate across three key scenarios: in-distribution, out-of-distribution, and combinatorial generalization. We developed a 2D simulation testbed for object movement and collisions to generate videos deterministically governed by one or more classical mechanics laws. This provides an unlimited supply of data for large-scale experimentation and enables quantitative evaluation of whether the generated videos adhere to physical laws. We trained diffusion-based video generation models to predict object movements based on initial frames. Our scaling experiments show perfect generalization within the distribution, measurable scaling behavior for combinatorial generalization, but failure in out-of-distribution scenarios. Further experiments reveal two key insights about the generalization mechanisms of these models: (1) the models fail to abstract general physical rules and instead exhibit "case-based" generalization behavior, i.e., mimicking the closest training example; (2) when generalizing to new cases, models are observed to prioritize different factors when referencing training data: color > size > velocity > shape. Our study suggests that scaling alone is insufficient for video generation models to uncover fundamental physical laws, despite its role in Sora's broader success. See our project page at https://phyworld.github.io

Motivation & Objective

• To evaluate whether video generation models can discover fundamental physical laws from visual data without human priors.
• To investigate the limitations of scaling video generation models in learning universal physical rules versus memorizing training examples.
• To systematically assess generalization across three scenarios: in-distribution, out-of-distribution, and combinatorial generalization.
• To analyze the underlying mechanisms of model generalization, particularly whether models abstract physical laws or rely on similarity-based retrieval.
• To determine whether visual representations alone are sufficient for accurate physics modeling in video generation.

Proposed method

• Developed a 2D simulation testbed that generates deterministic videos governed by classical mechanics laws (uniform motion, elastic collision, parabolic motion).
• Used the simulator to generate large-scale datasets (30K to 3M examples) with controlled variables for training diffusion-based video generation models.
• Designed a tool to infer object states (position, size) from generated video pixels for quantitative evaluation of physical consistency.
• Evaluated model performance across three generalization regimes: in-distribution (ID), out-of-distribution (OOD), and combinatorial generalization.
• Conducted scaling experiments by varying model size (22M to 310M parameters) and training data size (600K to 6M examples).
• Performed manual evaluation of test samples for physical plausibility and analyzed feature prioritization in generalization behavior.

Experimental results

Research questions

• RQ1 Can video generation models generalize to out-of-distribution scenarios without explicit supervision or human priors?
• RQ2 To what extent does scaling video generation models improve their ability to discover and apply universal physical laws?
• RQ3 What is the underlying mechanism of generalization in video generation models—abstract rule learning or case-based reasoning?
• RQ4 How do different visual attributes (color, size, velocity, shape) influence the model’s generalization behavior?
• RQ5 Is visual representation alone sufficient for accurate fine-grained physics modeling in video generation?

Key findings

• Video generation models achieve near-perfect in-distribution generalization across all tasks, regardless of model or data scale.
• Out-of-distribution generalization error remains high and does not improve with increased data or model size, indicating a fundamental limitation in rule abstraction.
• Combinatorial generalization improves significantly with scaling: abnormal predictions decreased from 67% to 10% as training data increased from 600K to 6M examples.
• Models exhibit case-based generalization, prioritizing the closest training example based on visual attributes in the order: color > size > velocity > shape.
• Visual ambiguity—such as subtle size differences or positional uncertainty at pixel level—leads to physically implausible predictions, even when videos appear visually realistic.
• The study concludes that scaling alone is insufficient for video generation models to discover fundamental physical laws, as they fail to generalize beyond memorized patterns.