[Paper Review] MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representations

Daniel Bogdoll, Yitian Yang+3 more|arXiv (Cornell University)|Nov 20, 2023

Autonomous Vehicle Technology and Safety4 citations

TL;DR

MUVO proposes a multimodal, unsupervised generative world model for autonomous driving that learns a sensor-agnostic 3D geometric voxel occupancy representation from raw camera and lidar data. By jointly predicting future camera images, lidar point clouds, and 3D occupancy grids in an action-conditioned manner, MUVO improves prediction quality across all modalities, with pre-trained geometric representations enhancing both visual and point cloud generation performance.

ABSTRACT

World models for autonomous driving have the potential to dramatically improve the reasoning capabilities of today's systems. However, most works focus on camera data, with only a few that leverage lidar data or combine both to better represent autonomous vehicle sensor setups. In addition, raw sensor predictions are less actionable than 3D occupancy predictions, but there are no works examining the effects of combining both multimodal sensor data and 3D occupancy prediction. In this work, we perform a set of experiments with a MUltimodal World Model with Geometric VOxel representations (MUVO) to evaluate different sensor fusion strategies to better understand the effects on sensor data prediction. We also analyze potential weaknesses of current sensor fusion approaches and examine the benefits of additionally predicting 3D occupancy.

Motivation & Objective

• To develop an unsupervised generative world model that leverages raw, high-resolution camera and lidar data for autonomous driving.
• To learn a sensor-agnostic 3D geometric representation of the environment using multimodal sensor inputs.
• To improve future prediction quality for both visual and point cloud observations through a shared geometric understanding.
• To enable downstream planning tasks by directly outputting 3D occupancy grids as a structured world representation.
• To investigate whether pre-training on camera and lidar alone can improve 3D occupancy prediction performance.

Proposed method

• The model processes raw RGB images and lidar point clouds through modality-specific encoders, followed by cross-modal feature fusion in a latent space.
• A transition model predicts future latent states conditioned on action sequences, enabling action-conditioned future prediction.
• The predicted latent states are decoded into three outputs: high-resolution RGB images, raw lidar point clouds, and 3D voxel occupancy grids.
• The 3D occupancy representation is learned as a geometric-aware, sensor-agnostic world state, enabling direct use in planning.
• Pre-training on camera and lidar data alone is used as a starting point to initialize the model, with ablation studies on fine-tuning and freezing weights.
• The framework uses a video diffusion-based decoder for high-resolution, temporally consistent generation of future observations.

Figure 1 : This example shows a high-resolution future prediction from MUVO for 3D occupancy and camera and lidar observations.

Experimental results

Research questions

• RQ1 Can a multimodal generative world model improve future prediction quality by learning a shared, geometric-aware 3D occupancy representation?
• RQ2 Does pre-training on raw camera and lidar data alone contribute meaningful spatial knowledge to 3D occupancy prediction?
• RQ3 Can the learned 3D occupancy representation enhance the quality of both camera image and lidar point cloud predictions?
• RQ4 How does the performance of 3D occupancy prediction vary when using pre-trained weights versus training from scratch?
• RQ5 Is action-conditioned 3D occupancy prediction feasible and beneficial in an unsupervised, multimodal setting?

Key findings

• The PTO (pre-trained, open weights) model outperformed the NPT (no pre-training) model in IoU and Recall for 3D occupancy prediction, indicating that pre-trained features improve geometric reasoning.
• The PTF (pre-trained, frozen) model showed gradual improvement in 3D occupancy performance over time, demonstrating that camera and lidar encoders already encode partial geometric information without explicit supervision.
• Incorporating 3D occupancy prediction led to slight but consistent improvements in Chamfer Distance for lidar point clouds and PSNR for RGB images, showing cross-modal knowledge transfer.
• The model performed better on the RL validation set than on the DS set for most metrics, except IoU, which was higher on DS due to more noise in predictions on sparse grids.
• The PTO model surpassed NPT in early training stages, but NPT eventually overtook in Precision, suggesting a more conservative prediction strategy when training from scratch.
• MUVO is the first to demonstrate action-conditioned 3D occupancy prediction in an unsupervised, multimodal setting using real-world sensor data.

Figure 2 : Overview of MUVO. Our model consists of three stages, from left to right. First, raw camera images and lidar point clouds are processed, encoded, and fused. The resulting latent representations are then fed into our transition model. Conditioned on actions, future states are predicted. Fi