[Paper Review] Visual Interaction Networks

Nicholas Watters, Andrea Tacchetti+4 more|arXiv (Cornell University)|Jun 5, 2017

Data Visualization and Analytics20 references69 citations

TL;DR

The Visual Interaction Network (VIN) learns to predict future object states from raw video by combining a CNN-based perceptual encoder with an Interaction Network-based dynamics predictor, enabling long-horizon physical forecasting including invisible objects.

ABSTRACT

From just a glance, humans can make rich predictions about the future state of a wide range of physical systems. On the other hand, modern approaches from engineering, robotics, and graphics are often restricted to narrow domains and require direct measurements of the underlying states. We introduce the Visual Interaction Network, a general-purpose model for learning the dynamics of a physical system from raw visual observations. Our model consists of a perceptual front-end based on convolutional neural networks and a dynamics predictor based on interaction networks. Through joint training, the perceptual front-end learns to parse a dynamic visual scene into a set of factored latent object representations. The dynamics predictor learns to roll these states forward in time by computing their interactions and dynamics, producing a predicted physical trajectory of arbitrary length. We found that from just six input video frames the Visual Interaction Network can generate accurate future trajectories of hundreds of time steps on a wide range of physical systems. Our model can also be applied to scenes with invisible objects, inferring their future states from their effects on the visible objects, and can implicitly infer the unknown mass of objects. Our results demonstrate that the perceptual module and the object-based dynamics predictor module can induce factored latent representations that support accurate dynamical predictions. This work opens new opportunities for model-based decision-making and planning from raw sensory observations in complex physical environments.

Motivation & Objective

• Provide a general-purpose model to predict future physical states from raw visual observations.
• Learn factored latent object representations that support accurate long-horizon dynamics.
• Demonstrate robustness to visual noise and partial observability across diverse physical systems.

Proposed method

• Use a CNN-based visual encoder to extract a state code from triplets of frames for each object.
• Employ an Interaction Network–based dynamics predictor with multiple temporal offsets to forecast next-step state codes.
• Decode state codes into object positions and velocities for training targets.
• Train end-to-end with a combined loss: prediction loss over future steps plus an auxiliary encoder loss.
• Evaluate rollouts over long horizons and compare against baselines including state-to-state and vision-only models.

Experimental results

Research questions

• RQ1 Can a perceptual front-end and an object-centered dynamics predictor jointly learn to infer states and predict future trajectories from video?
• RQ2 How well does VIN scale with the number of objects and with partially observable (invisible) objects?
• RQ3 Do temporal-offset aggregation and relational reasoning improve long-horizon physical predictions compared with baselines?
• RQ4 Is the model robust to visual encoder noise and capable of inferring hidden quantities like unseen masses?

Key findings

• VIN outperforms baselines on all datasets in inverse normalized loss across 3-object and 6-object scenes.
• VIN achieves accurate long-horizon rollouts, with Euclidean prediction error remaining low over 50 steps across datasets.
• VIN can infer invisible object positions (e.g., hidden springs) from the effects on visible objects, within about 4% of frame width for initial rollout steps.
• In drift (no interaction) scenarios, VIN’s performance aligns with an ablated version lacking the relation network, highlighting the role of relational reasoning when interactions exist.
• The perceptual/noisy input during training appears to improve long-rollout robustness compared to purely state-to-state models.