[Paper Review] Stochastic Prediction of Multi-Agent Interactions from Partial Observations

Chen Sun, Per Karlsson+3 more|arXiv (Cornell University)|Feb 25, 2019

Human Pose and Action Recognition49 references34 citations

TL;DR

We present Graph-VRNN, a graph-structured variational recurrent neural network that combines learned dynamics with partial visual observations to infer current states and forecast future states of multiple interacting agents. It outperforms baselines on basketball and soccer datasets.

ABSTRACT

We present a method that learns to integrate temporal information, from a learned dynamics model, with ambiguous visual information, from a learned vision model, in the context of interacting agents. Our method is based on a graph-structured variational recurrent neural network (Graph-VRNN), which is trained end-to-end to infer the current state of the (partially observed) world, as well as to forecast future states. We show that our method outperforms various baselines on two sports datasets, one based on real basketball trajectories, and one generated by a soccer game engine.

Motivation & Objective

• Motivate robust state estimation and future forecasting under partial observability of interacting agents.
• Develop a unified, end-to-end discriminative model that fuses a dynamics prior with visual evidence.
• Leverage graph-structured VRNNs to capture agent interactions and data association.
• Demonstrate improved state estimation and forecasting on basketball and soccer datasets.

Proposed method

• Use a VRNN per agent with a graph interaction network to model inter-agent dynamics.
• Condition the outputs on visual input via an attention-based decoder that blends current visuals and past beliefs.
• Share backbone features across agents with agent-specific identifiers to learn data association.
• Train with a variational objective (ELBO) augmented for forecasting; apply beta weighting and scheduled sampling.
• Incorporate heatmaps and latent sufficient statistics to capture uncertainty and facilitate future-state forecasting.

Experimental results

Research questions

• RQ1 Can a graph-structured VRNN jointly infer current multi-agent states and forecast future states from partially observed video frames?
• RQ2 Does incorporating stochastic latent dynamics and agent interactions improve state estimation and forecasting over non-graph or non-stochastic baselines?
• RQ3 How does partial observability (occlusions) affect tracking accuracy and predictive performance in basketball and soccer settings?
• RQ4 Does attention-based fusion of visual evidence and latent dynamics effectively “see beyond the pixels”?

Key findings

• Graph-VRNN improves state estimation for occluded agents compared to baselines like Visual only, RNN, VRNN, Indep-RNN, Social-RNN, and Graph-RNN.
• Adding stochastic latent dynamics yields better performance than purely deterministic models; graph-based interaction modeling outperforms simple pooling-based methods.
• In basketball data, Graph-VRNN achieves lower normalized L2 distances for both visible and hidden agents, and its forecasting loss improves over time.
• In soccer data, forecasting gains from stochastic Graph-RNN are present but smaller due to the simulator’s predictability, indicating dataset characteristics influence benefits of stochasticity.
• Qualitative results show belief states becoming more certain with observed frames and diversifying under future uncertainty in Graph-VRNN samples.