[Paper Review] Learning Multi-Object Dynamics with Compositional Neural Radiance Fields

Danny Driess, Zhiao Huang+3 more|arXiv (Cornell University)|Feb 24, 2022

Human Pose and Action Recognition19 citations

TL;DR

This paper proposes a compositional, object-centric dynamics model that uses Neural Radiance Fields (NeRFs) and graph neural networks (GNNs) to learn multi-object scene dynamics from multi-view images. By encoding each object into a 3D-aware latent vector via implicit encoders and reconstructing scenes through compositional NeRF rendering, the method enables stable, long-term dynamics prediction and generalization to novel object counts, outperforming non-compositional baselines in simulation and real-world experiments with rigid and deformable objects.

ABSTRACT

We present a method to learn compositional multi-object dynamics models from image observations based on implicit object encoders, Neural Radiance Fields (NeRFs), and graph neural networks. NeRFs have become a popular choice for representing scenes due to their strong 3D prior. However, most NeRF approaches are trained on a single scene, representing the whole scene with a global model, making generalization to novel scenes, containing different numbers of objects, challenging. Instead, we present a compositional, object-centric auto-encoder framework that maps multiple views of the scene to a set of latent vectors representing each object separately. The latent vectors parameterize individual NeRFs from which the scene can be reconstructed. Based on those latent vectors, we train a graph neural network dynamics model in the latent space to achieve compositionality for dynamics prediction. A key feature of our approach is that the latent vectors are forced to encode 3D information through the NeRF decoder, which enables us to incorporate structural priors in learning the dynamics models, making long-term predictions more stable compared to several baselines. Simulated and real world experiments show that our method can model and learn the dynamics of compositional scenes including rigid and deformable objects. Video: https://dannydriess.github.io/compnerfdyn/

Motivation & Objective

• To address the challenge of learning compositional, multi-object dynamics from image observations, especially when generalizing to scenes with different numbers of objects than seen during training.
• To incorporate strong 3D inductive biases into latent dynamics models by using NeRFs as decoders for object-specific latent vectors.
• To improve long-term prediction stability by enforcing 3D structure through NeRF-based reconstruction and using GNNs in the latent space with adaptive adjacency matrices.
• To enable effective planning in the latent space using RRT-based methods by ensuring stable and sharp image predictions.
• To demonstrate generalization to unseen object counts and complex dynamics, including deformable objects, in both simulated and real-world environments.

Proposed method

• An implicit object encoder maps multi-view images to a set of object-specific latent vectors, each encoding a distinct 3D object.
• Each latent vector parameterizes an individual NeRF, enabling 3D-aware scene reconstruction through compositional rendering from novel views.
• A graph neural network (GNN) dynamics model operates in the latent space, with edge features learned via an edge encoder and node updates via a node propagator network.
• The adjacency matrix between objects is estimated from the model's own predictions, enabling adaptive modeling of object interactions.
• The framework uses a differentiable rendering pipeline that enforces 3D consistency across views, improving latent vector quality and 3D generalization.
• A planning method based on RRTs is applied in the latent space, enabled by stable long-term predictions.

Experimental results

Research questions

• RQ1 Can a compositional, object-centric auto-encoder with NeRF decoders generalize dynamics prediction to scenes with different numbers of objects than seen during training?
• RQ2 Does incorporating 3D inductive biases through NeRF-based reconstruction improve long-term dynamics prediction stability compared to 2D image-based baselines?
• RQ3 Can a GNN-based dynamics model in the latent space, informed by predicted object interactions, achieve more accurate and stable long-term predictions than non-compositional models?
• RQ4 How well does the method generalize to real-world scenes with rigid and deformable objects, including novel object counts?
• RQ5 Does the use of individual NeRFs per object, rather than a global scene representation, improve performance even in scenes with a single object?

Key findings

• The method achieves significantly lower image reconstruction and prediction errors than non-compositional baselines, especially over long prediction horizons, with relative image error remaining low even after 15 prediction steps.
• In real-world experiments with a giraffe and shoe, the model maintains low image error (under 0.05) across 15 prediction steps, outperforming baselines that degrade rapidly.
• The model generalizes effectively to scenes with fewer objects than in training, such as scenes with only a shoe or a sand mold, where baselines fail due to distribution shift.
• Even in a deformable object experiment with a rope, the method maintains stable predictions, outperforming a dense GNN and CNN-based decoder variants, demonstrating robustness to shape changes.
• The use of individual NeRFs per object enables adaptive adjacency matrix learning, which improves performance even in single-object scenes, showing the benefit of compositional design.
• The framework enables successful RRT-based planning in the latent space due to stable, sharp image predictions, validating its utility for downstream control tasks.