[Paper Review] VRKitchen2.0-IndoorKit: A Tutorial for Augmented Indoor Scene Building in Omniverse

Yizhou Zhao, Steven Gong+3 more|arXiv (Cornell University)|Jun 23, 2022

Human Motion and Animation4 citations

TL;DR

This paper introduces IndoorKit, a Python-based toolkit integrated into NVIDIA Omniverse for building photo-realistic, physically-simulated indoor environments. It enables researchers to synthesize diverse indoor scenes, apply scene randomization, and integrate with machine learning models for embodied AI tasks such as robot control and avatar animation, all through a unified, extensible pipeline using USD and GPU-accelerated simulation.

ABSTRACT

With the recent progress of simulations by 3D modeling software and game engines, many researchers have focused on Embodied AI tasks in the virtual environment. However, the research community lacks a platform that can easily serve both indoor scene synthesis and model benchmarking with various algorithms. Meanwhile, computer graphics-related tasks need a toolkit for implementing advanced synthesizing techniques. To facilitate the study of indoor scene building methods and their potential robotics applications, we introduce INDOORKIT: a built-in toolkit for NVIDIA OMNIVERSE that provides flexible pipelines for indoor scene building, scene randomizing, and animation controls. Besides, combining Python coding in the animation software INDOORKIT assists researchers in creating real-time training and controlling avatars and robotics. The source code for this toolkit is available at https://github.com/realvcla/VRKitchen2.0-Tutorial, and the tutorial along with the toolkit is available at https://vrkitchen20-tutorial.readthedocs.io/en/

Motivation & Objective

• Address the lack of integrated platforms that support both indoor scene synthesis and benchmarking of embodied AI algorithms.
• Provide a flexible, extensible toolkit within Omniverse to streamline scene building, randomization, and simulation for robotics and AI research.
• Enable real-time training and control of avatars and robots using photo-realistic 3D assets and physics simulation.
• Facilitate interoperability between diverse 3D formats (e.g., 3D-Front, SAPIEN, iTHOR) and simulation pipelines via USD.
• Support the integration of machine learning models into simulation workflows for downstream tasks like navigation, manipulation, and animation.

Proposed method

• Utilizes NVIDIA Omniverse’s Python API to expose low-level simulation and rendering controls for scene construction and animation.
• Supports multiple indoor scene datasets (e.g., 3D-Front, iTHOR, SAPIEN) and enables custom dataset integration with metadata and visualization tools.
• Employs USD (Universal Scene Description) as a standardized, compressed format to store and transfer full scene data including models, textures, lights, and animations.
• Enables real-time physics simulation through support for rigid, soft, articulated, and fluid bodies using Omniverse’s physics engine.
• Integrates with deep learning frameworks (PyTorch, TensorFlow) via Python scripting to allow end-to-end training of neural policies in simulation.
• Provides built-in randomization strategies for scene backgrounds, materials, decorations, and lighting to enhance generalization in AI training.

Experimental results

Research questions

• RQ1 How can a unified toolkit streamline the creation of photo-realistic, physically accurate indoor environments for embodied AI research?
• RQ2 What are the key technical requirements for integrating 3D scene synthesis, physics simulation, and machine learning training in a single simulation platform?
• RQ3 How can scene randomization strategies improve the robustness and generalization of AI agents in simulated indoor environments?
• RQ4 What role does a standardized scene description format (USD) play in enabling interoperability across 3D modeling tools and simulation engines?
• RQ5 How can researchers efficiently integrate custom datasets and pre-trained models into a simulation pipeline for robotics and avatar control?

Key findings

• IndoorKit enables seamless integration of diverse 3D indoor scene datasets—including 3D-Front, iTHOR, and SAPIEN—into a single, extensible simulation pipeline using USD.
• Scene randomization strategies (background, material, decoration, and lighting) significantly enhance the diversity and realism of training environments.
• Real-time control of animatable avatars and robotic agents is achieved through integration with AMASS and Mixamo assets, enabling physics-accurate motion in complex scenes.
• GPU-accelerated physics and neural network training in Omniverse allow for low-latency, high-fidelity simulation of complex interactions.
• The toolkit supports end-to-end workflows from scene loading and labeling to model training and evaluation, reducing development overhead.
• Comprehensive documentation, tutorials, and open-source availability via GitHub and ReadTheDocs lower the barrier to entry for researchers in embodied AI and robotics.