[Paper Review] LOPR: Latent Occupancy PRediction using Generative Models

Bernard Lange, Masha Itkina+1 more|arXiv (Cornell University)|Oct 3, 2022

Autonomous Vehicle Technology and Safety4 citations

TL;DR

LOPR proposes a novel latent occupancy prediction framework that decouples representation learning from stochastic scene prediction using a VAE-GAN-based generative model and a transformer-based predictor in the learned latent space. It achieves state-of-the-art performance on NuScenes, Waymo Open, and a custom dataset by jointly conditioning on LiDAR, RGB cameras, and HD maps without manual labeling.

ABSTRACT

Environment prediction frameworks are integral for autonomous vehicles, enabling safe navigation in dynamic environments. LiDAR generated occupancy grid maps (L-OGMs) offer a robust bird's eye-view scene representation that facilitates joint scene predictions without relying on manual labeling unlike commonly used trajectory prediction frameworks. Prior approaches have optimized deterministic L-OGM prediction architectures directly in grid cell space. While these methods have achieved some degree of success in prediction, they occasionally grapple with unrealistic and incorrect predictions. We claim that the quality and realism of the forecasted occupancy grids can be enhanced with the use of generative models. We propose a framework that decouples occupancy prediction into: representation learning and stochastic prediction within the learned latent space. Our approach allows for conditioning the model on other available sensor modalities such as RGB-cameras and high definition maps. We demonstrate that our approach achieves state-of-the-art performance and is readily transferable between different robotic platforms on the real-world NuScenes, Waymo Open, and a custom dataset we collected on an experimental vehicle platform.

Motivation & Objective

• Address the limitations of deterministic, grid-cell-based occupancy prediction models that produce blurry, unrealistic outputs and fail to model scene-level stochasticity.
• Overcome the reliance on manually labeled data in trajectory prediction frameworks by enabling end-to-end, self-supervised learning from raw sensor inputs.
• Enable robust, transferable scene prediction across robotic platforms by decoupling representation learning from task-specific prediction.
• Improve realism and accuracy of future occupancy grid maps by modeling uncertainty and leveraging multimodal inputs (LiDAR, RGB, HD maps).

Proposed method

• Decompose occupancy prediction into two stages: (1) representation learning via a VAE-GAN model trained on multimodal sensor data (LiDAR, RGB, HD maps), and (2) stochastic prediction in the learned latent space.
• Train a VAE-GAN to learn a disentangled, low-dimensional latent representation of the scene that preserves spatial and semantic structure.
• Use a transformer-based decoder to generate stochastic future occupancy predictions conditioned on the latent code and multimodal context.
• Condition the prediction network on LiDAR occupancy grids, RGB images, and HD maps to improve scene understanding and generalization.
• Optimize the framework end-to-end using self-supervised sequence-to-sequence learning on unlabeled sequences of sensor data.
• Apply reparameterization and variational inference to enable differentiable sampling and backpropagation through stochastic predictions.

Experimental results

Research questions

• RQ1 Can a disentangled latent space learned from multimodal sensor data improve the realism and accuracy of future occupancy grid predictions?
• RQ2 Does decoupling representation learning from prediction enable better generalization and transferability across robotic platforms compared to end-to-end grid-based models?
• RQ3 Can stochastic prediction in the latent space outperform deterministic prediction in terms of scene-level realism and uncertainty modeling?
• RQ4 To what extent can the framework leverage RGB and HD map inputs to improve prediction under partial observability and occlusion?
• RQ5 Can the framework serve as a pre-training objective for downstream supervised tasks like trajectory prediction?

Key findings

• LOPR achieves state-of-the-art performance on the NuScenes, Waymo Open, and custom real-world datasets, outperforming prior L-OGM and video prediction models.
• The stochastic variant of LOPR with full multimodal conditioning (LiDAR, RGB, HD maps) achieves a 0.61 mIoU on the 3-second prediction horizon in NuScenes, significantly outperforming deterministic baselines.
• LOPR generates realistic, diverse predictions that correctly propagate moving agents and maintain static scene details, even in complex scenarios like oncoming vehicles and unusual road shapes.
• The framework successfully infers previously unobserved agents entering the scene, such as oncoming vehicles, demonstrating robust generalization and uncertainty modeling.
• LOPR shows strong transferability across platforms, enabling deployment on different robotic systems with minimal retraining.
• Qualitative results show that LOPR can generate plausible scene variations, including occlusion reasoning and dynamic interactions, even when the ground truth is not fully observed.