[Paper Review] MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Driving

Zirui Wu, Tianyu Liu+14 more|arXiv (Cornell University)|Jul 27, 2023

Advanced Neural Network Applications4 citations

TL;DR

MARS is a novel, open-source, NeRF-based simulator for autonomous driving that enables instance-aware, modular, and photorealistic scene rendering by separately modeling foreground objects and background environments. It achieves state-of-the-art performance in image realism (PSNR: 28.37 on V-KITTI) and supports flexible editing of object trajectories and appearances through its modular architecture with interchangeable neural radiance field backbones and sampling strategies.

ABSTRACT

Nowadays, autonomous cars can drive smoothly in ordinary cases, and it is widely recognized that realistic sensor simulation will play a critical role in solving remaining corner cases by simulating them. To this end, we propose an autonomous driving simulator based upon neural radiance fields (NeRFs). Compared with existing works, ours has three notable features: (1) Instance-aware. Our simulator models the foreground instances and background environments separately with independent networks so that the static (e.g., size and appearance) and dynamic (e.g., trajectory) properties of instances can be controlled separately. (2) Modular. Our simulator allows flexible switching between different modern NeRF-related backbones, sampling strategies, input modalities, etc. We expect this modular design to boost academic progress and industrial deployment of NeRF-based autonomous driving simulation. (3) Realistic. Our simulator set new state-of-the-art photo-realism results given the best module selection. Our simulator will be open-sourced while most of our counterparts are not. Project page: https://open-air-sun.github.io/mars/.

Motivation & Objective

• To address the challenge of simulating rare, high-risk corner cases in autonomous driving with minimal sim-to-real domain gap.
• To overcome limitations in existing simulators that lack instance-level control and photorealistic fidelity in dynamic scenes.
• To develop a modular framework enabling flexible integration of diverse NeRF backbones, sampling strategies, and input modalities for scalable research and deployment.
• To provide an open-source platform that supports instance editing and realistic sensor simulation for improved algorithm training.

Proposed method

• The simulator decomposes scenes into a background node and multiple foreground nodes, each modeled with separate neural radiance fields for independent control of static and dynamic properties.
• It uses ray-box intersection to determine visibility and processes rays through instance-specific coordinate frames before querying node representations.
• Foreground nodes use a category-level NeRF representation with learned embeddings for instance-specific attributes, while the background uses a grid-based or MLP-based NeRF for scene geometry.
• A modular design allows plug-and-play selection of different backbones (e.g., grid, MLP), samplers (e.g., proposal, coarse-to-fine), and supervision signals (e.g., depth, semantics, sky loss).
• The system renders RGB images, depth maps, and semantic masks by composing samples from all visible nodes along each ray.
• Training is supervised using multi-modal inputs including RGB images, poses, 3D bounding boxes, and optional depth and segmentation masks.

Experimental results

Research questions

• RQ1 Can a NeRF-based simulator achieve state-of-the-art photorealism in autonomous driving simulation while supporting instance-level editing?
• RQ2 How does modularity in NeRF components (backbones, samplers, supervision) affect rendering quality and flexibility?
• RQ3 To what extent can instance-aware modeling of dynamic vehicles and static backgrounds improve sim-to-real generalization?
• RQ4 Does the proposed framework outperform existing simulators in realism and controllability on public benchmarks?

Key findings

• The proposed MARS framework achieves state-of-the-art PSNR of 28.37 and SSIM of 0.907 on the V-KITTI benchmark, significantly outperforming prior methods.
• The ablation study confirms that the default setting (grid-based background, category-level foreground, coarse-to-fine sampling) yields the best performance, with PSNR 25.04 on KITTI and 28.37 on V-KITTI.
• The inclusion of sky loss and depth supervision improves rendering quality, reducing LPIPS by 0.046 on V-KITTI compared to baseline settings.
• The modular design enables effective ablation and comparison, with the best-performing configuration achieving a 2.5–3.5 dB PSNR gain over non-modular baselines.
• Instance editing is successfully demonstrated, including vehicle removal, addition, and trajectory modification, with realistic visual results.
• Despite high realism, training takes hours and real-time inference is not supported; dynamic specular effects on reflective surfaces remain a limitation.