[Paper Review] Predicting the dynamics of 2d objects with a deep residual network

François Fleuret|arXiv (Cornell University)|Oct 13, 2016

Robot Manipulation and Learning4 references3 citations

TL;DR

This paper proposes a deep residual network that predicts the dynamic evolution of 2D rigid shapes in a physics simulator using image-to-image regression. The model learns to detect the grasped object, infer motion including torque and collisions, and accurately reconstruct the final scene configuration from initial state and grasp point images, achieving high-fidelity prediction without explicit physics modeling.

ABSTRACT

We investigate how a residual network can learn to predict the dynamics of interacting shapes purely as an image-to-image regression task. With a simple 2d physics simulator, we generate short sequences composed of rectangles put in motion by applying a pulling force at a point picked at random. The network is trained with a quadratic loss to predict the image of the resulting configuration, given the image of the starting configuration and an image indicating the point of grasping. Experiments show that the network learns to predict accurately the resulting image, which implies in particular that (1) it segments rectangles as distinct components, (2) it infers which one contains the grasping point, (3) it models properly the dynamic of a single rectangle, including the torque, (4) it detects and handles collisions to some extent, and (5) it re-synthesizes properly the entire scene with displaced rectangles.

Motivation & Objective

• To investigate whether a residual network can learn to predict complex 2D object dynamics purely through end-to-end image-to-image regression.
• To assess the network's ability to infer physical properties such as object identity, motion, torque, and collision handling from visual inputs alone.
• To evaluate if the model can implicitly learn segmentation and scene re-synthesis without supervision on part boundaries or dynamics.
• To determine whether a simple, non-differentiable physics simulator can be effectively modeled by a deep learning architecture with minimal architectural tuning.

Proposed method

• The network uses a 18-layer residual architecture with 16-channel feature maps, employing 5×5 convolutional filters, batch normalization, ReLU activations, and skip connections to stabilize training.
• Input consists of two 64×64 grayscale images: the initial object configuration and a grasp-point indicator with a single white dot.
• The model is trained with a standard L2 loss between predicted and ground-truth final configurations, using stochastic gradient descent on 32,768 simulated sequences.
• The residual blocks are structured as residual modules with batch normalization and ReLU applied after the skip connection, enabling stable learning of deep representations.
• The network is trained end-to-end on a custom 2D physics simulator that includes elastic collisions, torque, and fluid friction, with all parameters fixed across sequences.
• Activations across layers are visualized to analyze internal representations, revealing channel-specific roles in detecting moving objects and segmenting the grasped shape.

Experimental results

Research questions

• RQ1 Can a deep residual network learn to predict the final configuration of multiple interacting 2D rigid bodies from an initial image and a grasp point location without explicit physics supervision?
• RQ2 To what extent can the network infer physical dynamics such as translation, rotation (torque), and collision responses from visual input alone?
• RQ3 Does the network implicitly learn to segment the grasped object and distinguish it from other objects in the scene?
• RQ4 How well does the model generalize to complex collision chains or edge cases where small changes in initial conditions lead to large outcome differences?
• RQ5 Can the network's internal representations reflect meaningful physical abstractions like object identity, motion, and force propagation?

Key findings

• The network achieves low test loss (down to 0.001355) and generalizes well, with no overfitting observed after 2,000 epochs of training.
• The model accurately predicts the final configuration, including correct translation, rotation (torque), and collision-induced motion propagation.
• Visual analysis of internal activations shows that specific feature channels specialize in detecting the grasped object, moving objects, and background, indicating implicit segmentation.
• The model handles collisions robustly in many cases, though performance degrades on complex, multi-step collision chains or near-boundary scenarios.
• Even with minimal architectural tuning and no explicit physics loss, the network learns to re-synthesize the entire scene with high fidelity, including edge reconstruction and object displacement.
• Reducing the number of channels from 16 to 8 significantly degrades performance, indicating that sufficient capacity is essential for learning complex dynamics.