[Paper Review] ReSimAD: Zero-Shot 3D Domain Transfer for Autonomous Driving with Source Reconstruction and Target Simulation

Bo Zhang, Xinyu Cai+11 more|arXiv (Cornell University)|Sep 11, 2023

Domain Adaptation and Few-Shot LearningComputer Science3 citations

TL;DR

ReSimAD proposes a zero-shot 3D domain adaptation framework for autonomous driving that reconstructs 3D scenes from source-domain annotated data and simulates target-domain-like point clouds by injecting target-specific sensor characteristics. The method achieves state-of-the-art zero-shot 3D detection performance on cross-domain benchmarks like Waymo-to-KITTI and Waymo-to-nuScenes, even surpassing supervised domain adaptation baselines with real target data, and enables effective 3D pre-training using simulated data.

ABSTRACT

Domain shifts such as sensor type changes and geographical situation variations are prevalent in Autonomous Driving (AD), which poses a challenge since AD model relying on the previous domain knowledge can be hardly directly deployed to a new domain without additional costs. In this paper, we provide a new perspective and approach of alleviating the domain shifts, by proposing a Reconstruction-Simulation-Perception (ReSimAD) scheme. Specifically, the implicit reconstruction process is based on the knowledge from the previous old domain, aiming to convert the domain-related knowledge into domain-invariant representations, e.g., 3D scene-level meshes. Besides, the point clouds simulation process of multiple new domains is conditioned on the above reconstructed 3D meshes, where the target-domain-like simulation samples can be obtained, thus reducing the cost of collecting and annotating new-domain data for the subsequent perception process. For experiments, we consider different cross-domain situations such as Waymo-to-KITTI, Waymo-to-nuScenes, Waymo-to-ONCE, etc, to verify the zero-shot target-domain perception using ReSimAD. Results demonstrate that our method is beneficial to boost the domain generalization ability, even promising for 3D pre-training.

Motivation & Objective

• Address the challenge of domain shift in autonomous driving due to sensor type changes or geographical variations.
• Enable zero-shot transfer of 3D perception models to new domains without collecting or annotating real target-domain data.
• Reduce reliance on costly real-world data collection and human annotation for model deployment in new environments.
• Explore the feasibility of using reconstructed and simulated data for 3D pre-training to improve generalization.

Proposed method

• Reconstruct 3D scene-level meshes from source-domain annotated point clouds using implicit reconstruction, decoupling domain-specific features.
• Condition target-domain simulation on the reconstructed 3D meshes, injecting target-specific LiDAR parameters and scene attributes.
• Generate synthetic point clouds that mimic real target-domain data distribution, preserving structural and semantic realism.
• Utilize the simulated data for zero-shot 3D detection and pre-training, avoiding real data collection and annotation.
• Integrate the simulated data into standard 3D detection pipelines, such as PV-RCNN++ and SECOND-IOU, for downstream fine-tuning.
• Leverage minimal prior knowledge about the target domain (e.g., sensor specs) to guide the simulation process.

Experimental results

Research questions

• RQ1 Can implicit 3D reconstruction from source-domain data produce domain-invariant scene representations suitable for zero-shot transfer?
• RQ2 Can simulating target-domain-like point clouds from reconstructed 3D meshes achieve high detection performance on unseen domains without real data?
• RQ3 How effective is ReSimAD compared to supervised domain adaptation methods that use real target-domain data?
• RQ4 Can ReSimAD-generated data support effective 3D pre-training for downstream perception tasks?

Key findings

• ReSimAD achieves a 36.45 / 18.94 BEV/AP3D on Waymo-to-nuScenes zero-shot detection, outperforming the zero-shot baseline (24.57 / 15.12) and approaching the oracle performance (50.54 / 33.41).
• On Waymo-to-KITTI, ReSimAD achieves 70.57 / 84.50 / 57.06 APBEV/AP3D, demonstrating strong zero-shot generalization without real target data.
• Using 100K simulated samples, ReSimAD enables 3D pre-training that achieves comparable or better fine-tuning performance than pre-training on 100K real ONCE data.
• The method shows consistent gains on lightweight detectors like SECOND-IOU, indicating practical applicability in real-world systems.
• Visualizations confirm that ReSimAD-generated point clouds preserve realistic scene structures, such as road surfaces and street layouts, outperforming direct CARLA simulation.
• The implicit reconstruction process produces clearer, more continuous meshes than prior methods like VDBFusion, enhancing simulation fidelity.