[Paper Review] SurfelGAN: Synthesizing Realistic Sensor Data for Autonomous Driving

Zhenpei Yang, Yuning Chai+6 more|arXiv (Cornell University)|May 8, 2020

Computer Graphics and Visualization Techniques40 references4 citations

TL;DR

SurfelGAN proposes a data-driven approach to synthesize realistic camera images for autonomous driving simulation using only real lidar and camera data from a single vehicle pass. It reconstructs scenes with textured surfels, then employs a GAN to generate high-fidelity images for novel viewpoints, significantly improving realism and enabling effective data augmentation for object detection.

ABSTRACT

Autonomous driving system development is critically dependent on the ability to replay complex and diverse traffic scenarios in simulation. In such scenarios, the ability to accurately simulate the vehicle sensors such as cameras, lidar or radar is essential. However, current sensor simulators leverage gaming engines such as Unreal or Unity, requiring manual creation of environments, objects and material properties. Such approaches have limited scalability and fail to produce realistic approximations of camera, lidar, and radar data without significant additional work. In this paper, we present a simple yet effective approach to generate realistic scenario sensor data, based only on a limited amount of lidar and camera data collected by an autonomous vehicle. Our approach uses texture-mapped surfels to efficiently reconstruct the scene from an initial vehicle pass or set of passes, preserving rich information about object 3D geometry and appearance, as well as the scene conditions. We then leverage a SurfelGAN network to reconstruct realistic camera images for novel positions and orientations of the self-driving vehicle and moving objects in the scene. We demonstrate our approach on the Waymo Open Dataset and show that it can synthesize realistic camera data for simulated scenarios. We also create a novel dataset that contains cases in which two self-driving vehicles observe the same scene at the same time. We use this dataset to provide additional evaluation and demonstrate the usefulness of our SurfelGAN model.

Motivation & Objective

• Address the challenge of creating scalable, realistic sensor simulations for autonomous driving without relying on manual environment design in gaming engines.
• Overcome limitations of traditional simulators that require extensive manual setup of materials, geometry, and sensor models.
• Develop a purely data-driven pipeline that leverages real-world sensor data to generate high-fidelity visual outputs for novel vehicle trajectories.
• Enable realistic data augmentation for training perception models by synthesizing diverse, photorealistic views of real scenes.
• Create a novel benchmark dataset with dual-vehicle observations to evaluate novel view synthesis under realistic multi-agent conditions.

Proposed method

• Reconstruct 3D scenes from single or multi-pass autonomous vehicle data using an enhanced surfel representation that encodes geometry, texture, semantics, and appearance.
• Render novel views of the scene from new self-driving vehicle poses and object configurations using the surfel map.
• Apply a GAN-based image synthesis network (SurfelGAN) to transform low-fidelity surfel renderings into photorealistic images.
• Train SurfelGAN using a combination of real images, surfel renderings, and a novel internal camera dataset, optimizing for perceptual realism and L1 distance to real images.
• Use a cycle-consistency loss and adversarial training to improve image quality and reduce artifacts in generated outputs.
• Apply the model to generate data for data augmentation, projecting 3D bounding boxes from real data to 2D novel views for detector training.

Experimental results

Research questions

• RQ1 Can a data-driven approach generate photorealistic camera images for autonomous driving scenarios without manual environment creation?
• RQ2 To what extent can surfel-based scene reconstruction and GAN-based image synthesis produce images that match real sensor data in visual quality and statistical properties?
• RQ3 How effective is SurfelGAN-generated data for data augmentation in training object detectors compared to real data or surfel renderings alone?
• RQ4 What are the failure modes of the system, and how do reconstruction errors or missing geometry affect image synthesis quality?
• RQ5 Can a novel dataset with dual-vehicle observations improve the evaluation of novel view synthesis in autonomous driving simulation?

Key findings

• SurfelGAN significantly improves image realism over surfel renderings, reducing L1 distance to real images and outperforming models with additional losses and data.
• The SurfelGAN-S variant, which optimizes for L1 distance, achieves better performance than more complex models (SA and SAC), demonstrating the effectiveness of direct optimization on the target metric.
• Data augmentation using SurfelGAN-generated images boosts the average precision of a vehicle detector by 3.5 percentage points (AP@50 from 21.9% to 25.4%) compared to training on real data alone.
• SurfelGAN-generated images lead to a 1.3% improvement in AP@75 (from 10.8% to 12.1%) and a 1.1% increase in average AP (from 11.9% to 13.0%), showing strong practical utility.
• Failure cases reveal limitations in reconstruction quality, particularly in unmapped regions or areas with large geometric errors, where the GAN struggles to hallucinate realistic textures.
• The proposed dataset with dual-vehicle observations enables reliable evaluation of novel view synthesis, providing a new benchmark for future research in autonomous driving simulation.