[Paper Review] GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Driving

Yun Chen, Frieda Rong+7 more|arXiv (Cornell University)|Jan 16, 2021

Computer Graphics and Visualization Techniques60 references4 citations

TL;DR

GeoSim proposes a geometry-aware video simulation framework that composites realistic 3D dynamic objects—extracted from sensor data and rendered at novel poses—into existing urban scenes to generate photorealistic, geometrically consistent, and traffic-aware synthetic videos. The method achieves state-of-the-art perceptual quality and boosts semantic segmentation performance by 3.4% mIOU via data augmentation.

ABSTRACT

Scalable sensor simulation is an important yet challenging open problem for safety-critical domains such as self-driving. Current works in image simulation either fail to be photorealistic or do not model the 3D environment and the dynamic objects within, losing high-level control and physical realism. In this paper, we present GeoSim, a geometry-aware image composition process which synthesizes novel urban driving scenarios by augmenting existing images with dynamic objects extracted from other scenes and rendered at novel poses. Towards this goal, we first build a diverse bank of 3D objects with both realistic geometry and appearance from sensor data. During simulation, we perform a novel geometry-aware simulation-by-composition procedure which 1) proposes plausible and realistic object placements into a given scene, 2) render novel views of dynamic objects from the asset bank, and 3) composes and blends the rendered image segments. The resulting synthetic images are realistic, traffic-aware, and geometrically consistent, allowing our approach to scale to complex use cases. We demonstrate two such important applications: long-range realistic video simulation across multiple camera sensors, and synthetic data generation for data augmentation on downstream segmentation tasks. Please check https://tmux.top/publication/geosim/ for high-resolution video results.

Motivation & Objective

• To address the lack of scalable, photorealistic, and physically consistent video simulation for self-driving systems.
• To eliminate the need for manual 3D asset creation by leveraging large-scale sensor data from real-world driving fleets.
• To enable realistic insertion of dynamic objects (e.g., vehicles) into existing scenes with proper occlusion, lighting, and spatial consistency.
• To demonstrate the utility of synthetic data in improving downstream perception tasks like semantic segmentation.
• To achieve high realism and geometric fidelity in video simulation across multiple camera sensors.

Proposed method

• Constructs a large-scale 3D asset bank from multi-view, multi-sensor data (LiDAR, cameras) using a differentiable reconstruction network with shape regularization.
• Uses HD maps and LiDAR to infer plausible 3D object placements in the scene, ensuring traffic-aware and physically plausible positioning.
• Performs image-based rendering with neural inpainting to handle occlusions, lighting variations, and boundary blending for seamless composition.
• Employs a hybrid rendering pipeline combining physics-based rendering and neural inpainting to enhance realism while maintaining efficiency.
• Generates temporally consistent video sequences across multiple camera sensors by propagating object poses and rendering novel views.
• Applies the synthetic data for data augmentation in semantic segmentation, fine-tuning models on real data with added synthetic examples.

Experimental results

Research questions

• RQ1 Can a data-driven, geometry-aware composition method generate synthetic driving videos that are perceptually indistinguishable from real footage?
• RQ2 How does the integration of 3D scene layout and dynamic object placement improve realism and geometric consistency in video simulation?
• RQ3 To what extent can GeoSim-generated synthetic data improve the performance of downstream perception models like semantic segmentation?
• RQ4 What is the relative contribution of shadow rendering and hybrid rendering to perceived realism, despite limited FID score improvements?
• RQ5 Can GeoSim generalize across different datasets and urban environments without retraining?

Key findings

• GeoSim achieves a human preference score of 94.3% over baselines in perceptual quality evaluation, significantly outperforming methods like SPADE, Guided Editing, and CAD.
• The FID score of GeoSim is 14.3, which is lower than all baselines, indicating superior distributional similarity to real data.
• Including shadows in the rendering pipeline increases human preference from 71.9% to - (implied improvement), though FID remains unchanged at 14.3, highlighting a gap between human perception and FID metrics.
• On the Argoverse dataset, GeoSim achieves a human preference score of 84.0% over CAD, with a FID of 24.5, demonstrating strong generalization across datasets.
• Data augmentation using GeoSim boosts mIOU by 3.4% on PSPNet and 0.4% on DeepLabv3, with consistent gains across two segmentation models.
• The method reduces artifacts like smeared cars and geometric inconsistencies seen in prior methods, achieving seamless blending through neural inpainting and occlusion handling.