[Paper Review] ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation

Chuang Gan, J. Schwartz+22 more|arXiv (Cornell University)|Jul 9, 2020

Generative Adversarial Networks and Image Synthesis42 references129 citations

TL;DR

TDW is a general-purpose virtual world platform that combines near-photorealistic rendering, multi-material physics (including rigid, soft, cloth, and fluids), audio synthesis, and VR-enabled human/agent interaction to create rich multi-modal simulations for AI research.

ABSTRACT

We introduce ThreeDWorld (TDW), a platform for interactive multi-modal physical simulation. TDW enables simulation of high-fidelity sensory data and physical interactions between mobile agents and objects in rich 3D environments. Unique properties include: real-time near-photo-realistic image rendering; a library of objects and environments, and routines for their customization; generative procedures for efficiently building classes of new environments; high-fidelity audio rendering; realistic physical interactions for a variety of material types, including cloths, liquid, and deformable objects; customizable agents that embody AI agents; and support for human interactions with VR devices. TDW's API enables multiple agents to interact within a simulation and returns a range of sensor and physics data representing the state of the world. We present initial experiments enabled by TDW in emerging research directions in computer vision, machine learning, and cognitive science, including multi-modal physical scene understanding, physical dynamics predictions, multi-agent interactions, models that learn like a child, and attention studies in humans and neural networks.

Motivation & Objective

• Motivate and enable training and benchmarking of embodied AI and cognitive science models in a single, multi-modal simulation environment.
• Provide high-fidelity visual and auditory rendering along with advanced physics to support diverse interaction scenarios.
• Offer a flexible API and asset library to create varied, controllable environments for perception, navigation, and manipulation tasks.
• Demonstrate TDW's utility through cross-domain experiments in vision, audio, physics prediction, multi-agent interaction, and human-robot collaboration.
• Facilitate future research by enabling synthetic data generation with controllable physical parameters not easily labeled in the real world.

Proposed method

• Integrates Unity-based rendering with two physics engines (NVIDIA Flex for non-rigid/material interactions and PhysX for rigid bodies).
• Provides a two-component architecture: Build (Unity-based) for rendering/audio/physics and Controller (Python API) for task specification; supports over 200 commands and single-timestep command batching.
• Uses procedurally generated 3D assets and environments with PBR materials and a 2,500-object library to enable scalable scene creation.
• Introduces PyImpact for real-time, physics-driven impact sound synthesis, and uses Resonance Audio for 3D spatial audio.
• Supports three interaction paradigms: direct API control of objects, embodied AI agents (including a robotics-like Magnebot and URDF-imported bots), and VR-human interaction.
• Presents multi-modal experiments including visual feature transfer, audio-based material classification, and multimodal scene understanding.

Experimental results

Research questions

• RQ1 Can TDW-generated data train representations that transfer effectively to real-world vision and audio tasks?
• RQ2 How well do physics-informed multi-modal simulations support learning of material properties and mass from audiovisual cues?
• RQ3 Can end-to-end differentiable physical predictions be learned using TDW-generated data and architectures like Hierarchical Relation Networks (HRN)?
• RQ4 What is the utility of TDW for studying multi-agent interactions and attention in VR, and how do human vs. neural agents compare?
• RQ5 How does TDW enable scalable, controllable experimentation across perception, dynamics, and social interaction domains?

Key findings

• TDW-generated visual representations transfer to fine-grained classification tasks with performance approaching ImageNet-pretrained models in several categories.
• Audio-based material classification trained on TDW synthetic sounds significantly outperforms a Sound-20K baseline, indicating richer synthetic audio diversity improves generalization.
• Multimodal (vision+audio) inputs yield higher accuracy for material and mass classification than either modality alone, underscoring the value of realistic multi-modal rendering.
• Learnable physical predictors trained on TDW data (DRHRN) outperform baselines in predicting dynamics across lift, slide, collide, stack, and cloth scenarios, demonstrating improved generalization and shape preservation.
• The platform enables advanced physical scene understanding benchmarks (e.g., object permanence, draping, submerging) and supports learning-based dynamics models.