[Paper Review] Flexible Neural Representation for Physics Prediction

Damian Mrowca, Chengxu Zhuang+5 more|arXiv (Cornell University)|Jun 21, 2018

Human Pose and Action Recognition46 references86 citations

TL;DR

The paper introduces Hierarchical Relation Network (HRN) with a hierarchical particle graph to predict complex physics, including rigid and deformable objects, achieving long-horizon, plausible dynamics and strong generalization beyond training scenarios.

ABSTRACT

Humans have a remarkable capacity to understand the physical dynamics of objects in their environment, flexibly capturing complex structures and interactions at multiple levels of detail. Inspired by this ability, we propose a hierarchical particle-based object representation that covers a wide variety of types of three-dimensional objects, including both arbitrary rigid geometrical shapes and deformable materials. We then describe the Hierarchical Relation Network (HRN), an end-to-end differentiable neural network based on hierarchical graph convolution, that learns to predict physical dynamics in this representation. Compared to other neural network baselines, the HRN accurately handles complex collisions and nonrigid deformations, generating plausible dynamics predictions at long time scales in novel settings, and scaling to large scene configurations. These results demonstrate an architecture with the potential to form the basis of next-generation physics predictors for use in computer vision, robotics, and quantitative cognitive science.

Motivation & Objective

• Motivate a flexible, object-centric representation that can capture arbitrary rigid and nonrigid materials.
• Develop a scalable neural architecture to predict physical dynamics across complex multi-object scenes.
• Enable long-horizon predictions and generalization to unseen shapes and interactions.
• Incorporate learned material properties and hierarchical reasoning to improve accuracy over baselines.

Proposed method

• Represent each object as a set of particles forming a hierarchical graph to capture shape and material variation.
• Use a three-stage hierarchical graph convolution (Leaves-to-Ancestors, Within-Siblings, Ancestors-to-Descendants) to propagate effects efficiently.
• Separate neural modules for external forces (phi_F), collisions (phi_C), and history (phi_H), followed by a hierarchical propagation (eta) and a state predictor (psi).
• Train with a composite loss combining local (per-particle), global (whole-object), and distance-preservation terms to enforce physical constraints.
• Generate training data with a particle-based environment (NVIDIA FleX in Unity3D) and learn material relationships from data rather than hard-coded physics.

Experimental results

Research questions

• RQ1 Can HRN predict future states for scenes with rigid, soft, and deformable bodies including complex collisions?
• RQ2 Does the hierarchical graph structure improve accuracy and scalability compared to flat graphs or MLP baselines?
• RQ3 How well does HRN generalize to unseen shapes, materials, and multi-object interactions?
• RQ4 What is the impact of the loss components (local/global/preservation) and multi-step inputs on predictive accuracy?

Key findings

• HRN accurately handles complex collisions and nonrigid deformations and generates plausible long-horizon dynamics.
• The hierarchical representation and three-stage propagation outperform baselines including MLP, fully-connected graphs, and non-hierarchical models.
• Ablations show the importance of hierarchy, separate modules, and the preservation/global losses for prediction quality.
• HRN generalizes to unseen shapes, materials, and multi-object scenarios, including towers and domino-like setups.