[Paper Review] Neural Relational Inference for Interacting Systems

Thomas Kipf, Ethan Fetaya+3 more|arXiv (Cornell University)|Feb 13, 2018

Human Pose and Action Recognition45 references255 citations

TL;DR

An unsupervised variational autoencoder that jointly infers latent interaction graphs and learns dynamics via graph neural networks, enabling interpretable edge types and accurate future-state predictions on simulated and real-world interacting systems.

ABSTRACT

Interacting systems are prevalent in nature, from dynamical systems in physics to complex societal dynamics. The interplay of components can give rise to complex behavior, which can often be explained using a simple model of the system's constituent parts. In this work, we introduce the neural relational inference (NRI) model: an unsupervised model that learns to infer interactions while simultaneously learning the dynamics purely from observational data. Our model takes the form of a variational auto-encoder, in which the latent code represents the underlying interaction graph and the reconstruction is based on graph neural networks. In experiments on simulated physical systems, we show that our NRI model can accurately recover ground-truth interactions in an unsupervised manner. We further demonstrate that we can find an interpretable structure and predict complex dynamics in real motion capture and sports tracking data.

Motivation & Objective

• Motivate learning underlying interactions in dynamical systems from trajectories without manual edge annotations.
• Propose a neural relational inference (NRI) model that jointly learns edge-type interactions and system dynamics.
• Enable interpretable, discrete edge-types representing interactions via a probabilistic graph decoder.
• Demonstrate unsupervised recovery of ground-truth interactions and accurate long-horizon predictions on simulated physics, motion capture, and sports data.

Proposed method

• Encode observed trajectories with a GNN over a fully-connected graph to infer edge-type distributions q_phi(z_ij|x).
• Use a discrete latent graph (edge types) represented as one-hot vectors and a continuous relaxation (concrete distribution) for reparameterization.
• Decode future trajectories with a GNN-based decoder conditioned on the inferred graph z, with separate edge-type decoders to model distinct interactions.
• Train with a variational objective (ELBO) combining a reconstruction term E_{q_phi(z|x)}[log p_theta(x|z)] and a KL term KL[q_phi(z|x)||p_theta(z)].
• Mitigate degenerate decoding by predicting multiple future steps and using per-edge-type decoders to enforce dependence on edge types.

Experimental results

Research questions

• RQ1 Can a model infer latent interaction graphs from trajectories without supervision?
• RQ2 How well can the inferred edge-types correspond to ground-truth interactions in physical simulations?
• RQ3 Do edge-type aware decoders improve long-horizon predictive accuracy over fully connected or non-relational baselines?
• RQ4 Is the approach robust to real-world data such as motion capture and sports tracking, yielding interpretable interaction structure?

Key findings

• The NRI model accurately recovers ground-truth interaction graphs in unsupervised physical system experiments.
• NRI learns a small number of edge types that enable accurate long-horizon dynamics prediction on motion capture and sports data.
• Dynamic re-evaluation of the latent graph improves predictive performance on real motion capture data.
• Edge-type specific decoders and multi-step prediction mitigate degenerate decoding and enhance learning of interactions.
• On multiple simulated tasks, NRI learned graphs approach or match supervised and gold-standard baselines for interaction recovery and prediction.