[Paper Review] SPACE: A Simulator for Physical Interactions and Causal Learning in 3D Environments

Jiafei Duan, Samson Yu+1 more|arXiv (Cornell University)|Aug 13, 2021

Human Pose and Action Recognition31 references4 citations

TL;DR

This paper introduces SPACE, a 3D simulator and synthetic video dataset for fundamental physical interactions—containment, stability, and contact—designed to improve causal reasoning in AI. By pre-training a physics-based model (PhyDNet) on SPACE using a curriculum learning approach, the model achieves significantly lower validation MSE on real-world action datasets, demonstrating that learning basic physical interactions enhances robustness in intuitive physics reasoning.

ABSTRACT

Recent advancements in deep learning, computer vision, and embodied AI have given rise to synthetic causal reasoning video datasets. These datasets facilitate the development of AI algorithms that can reason about physical interactions between objects. However, datasets thus far have primarily focused on elementary physical events such as rolling or falling. There is currently a scarcity of datasets that focus on the physical interactions that humans perform daily with objects in the real world. To address this scarcity, we introduce SPACE: A Simulator for Physical Interactions and Causal Learning in 3D Environments. The SPACE simulator allows us to generate the SPACE dataset, a synthetic video dataset in a 3D environment, to systematically evaluate physics-based models on a range of physical causal reasoning tasks. Inspired by daily object interactions, the SPACE dataset comprises videos depicting three types of physical events: containment, stability and contact. These events make up the vast majority of the basic physical interactions between objects. We then further evaluate it with a state-of-the-art physics-based deep model and show that the SPACE dataset improves the learning of intuitive physics with an approach inspired by curriculum learning. Repository: https://github.com/jiafei1224/SPACE

Motivation & Objective

• To address the lack of synthetic datasets focusing on everyday physical interactions in 3D environments, such as containment, stability, and contact.
• To develop a physics-based 3D simulator (SPACE) capable of generating diverse, realistic physical interactions with accurate dynamics.
• To create a large-scale video dataset (SPACE dataset) with over 15,000 unique videos and 2 million frames across three fundamental physical interaction types.
• To evaluate whether pre-training on the SPACE dataset improves the performance of physics-based deep learning models on real-world action prediction tasks.
• To validate the effectiveness of a curriculum learning approach by pre-training on fundamental physical interactions before fine-tuning on real-world datasets.

Proposed method

• The SPACE simulator generates 3D physical interactions using physics-based simulation with configurable object geometries, materials, and environmental conditions.
• The simulator produces videos with multiple modalities: RGB, object segmentation, optical flow, depth, and surface normal vectors for each frame.
• The SPACE dataset comprises 15,000 unique videos across three physical interaction types: containment (e.g., objects inside containers), stability (e.g., stacked objects), and contact (e.g., pushing or touching).
• A state-of-the-art physics-based model, PhyDNet, is pre-trained on the SPACE dataset using teacher forcing and adaptive learning rate scheduling.
• Fine-tuning is performed on the UCF101 dataset using a curriculum learning-inspired setup, where models are first pre-trained on SPACE and then fine-tuned on real-world action videos.
• Evaluation uses Mean Squared Error (MSE) averaged over 45 predicted frames to compare generated frames against ground truth.

Experimental results

Research questions

• RQ1 Can a synthetic 3D simulator generate diverse, physically accurate interactions representative of everyday human-object interactions?
• RQ2 Does pre-training on fundamental physical interactions (containment, stability, contact) improve the performance of physics-based models on real-world video prediction tasks?
• RQ3 Is the contribution of each individual physical interaction type (containment, stability, contact) essential for learning robust physical dynamics?
• RQ4 To what extent does curriculum learning—pre-training on simple physical interactions before fine-tuning on complex real-world actions—improve model generalization?
• RQ5 How does the SPACE dataset compare to existing datasets like CLEVRER, CoPhy, and CATER in terms of physical interaction complexity and model performance?

Key findings

• Pre-training on the SPACE dataset significantly reduces validation loss on the UCF101 dataset, achieving the lowest MSE after 30 epochs of fine-tuning.
• The experimental setup with pre-training on all three SPACE tasks (containment, stability, contact) outperforms individual task pre-training, indicating that all three are essential for robust physical dynamics learning.
• Models pre-trained on SPACE achieved lower validation MSE than those pre-trained only on UCF101, demonstrating the benefit of synthetic data for physical reasoning.
• Individual pre-training on each of the three SPACE tasks (containment, stability, contact) yielded worse performance than combined pre-training, confirming the complementary nature of the three interaction types.
• The curriculum learning approach—pre-training on SPACE before fine-tuning on UCF101—led to faster convergence and better generalization on real-world action prediction tasks.
• The SPACE dataset enables PhyDNet to learn more accurate physical dynamics, as evidenced by lower MSE across all fine-tuning scenarios.