[Paper Review] Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots

Xavier Puig, Eric Undersander+21 more|arXiv (Cornell University)|Oct 19, 2023

Social Robot Interaction and HRIPsychology15 citations

TL;DR

Habitat 3.0 introduces a fast, realistic simulator that supports humanoid avatars and robots for collaborative human–robot tasks in home-like environments, plus a human-in-the-loop evaluation tool and two tasks (Social Navigation and Social Rearrangement) with multiple baselines.

ABSTRACT

We present Habitat 3.0: a simulation platform for studying collaborative human-robot tasks in home environments. Habitat 3.0 offers contributions across three dimensions: (1) Accurate humanoid simulation: addressing challenges in modeling complex deformable bodies and diversity in appearance and motion, all while ensuring high simulation speed. (2) Human-in-the-loop infrastructure: enabling real human interaction with simulated robots via mouse/keyboard or a VR interface, facilitating evaluation of robot policies with human input. (3) Collaborative tasks: studying two collaborative tasks, Social Navigation and Social Rearrangement. Social Navigation investigates a robot's ability to locate and follow humanoid avatars in unseen environments, whereas Social Rearrangement addresses collaboration between a humanoid and robot while rearranging a scene. These contributions allow us to study end-to-end learned and heuristic baselines for human-robot collaboration in-depth, as well as evaluate them with humans in the loop. Our experiments demonstrate that learned robot policies lead to efficient task completion when collaborating with unseen humanoid agents and human partners that might exhibit behaviors that the robot has not seen before. Additionally, we observe emergent behaviors during collaborative task execution, such as the robot yielding space when obstructing a humanoid agent, thereby allowing the effective completion of the task by the humanoid agent. Furthermore, our experiments using the human-in-the-loop tool demonstrate that our automated evaluation with humanoids can provide an indication of the relative ordering of different policies when evaluated with real human collaborators. Habitat 3.0 unlocks interesting new features in simulators for Embodied AI, and we hope it paves the way for a new frontier of embodied human-AI interaction capabilities.

Motivation & Objective

• Motivate and enable study of social embodied AI by simulating cooperative human-robot tasks in realistic home environments.
• Offer fast, diverse humanoid appearance and motion models suitable for learning-based agents.
• Provide a human-in-the-loop evaluation interface to assess robot policies with real human collaborators.
• Introduce two social tasks (Social Navigation and Social Rearrangement) with standardized baselines for benchmarking.
• Demonstrate generalization and emergent cooperative behaviors in human-robot collaboration.

Proposed method

• Humanoid appearance and motion: use a dual representation with an articulated skeleton for physics and a skinned mesh for rendering, based on SMPL-X; cache multiple body shapes to accelerate loading and pose generation.
• Humanoid motion: hierarchical policy that composes low-level skills (navigation, pick/place) with a path planner and motion clips (walking cycle) to achieve long-range behaviors.
• Human-in-the-loop (HITL) tool: a client-server setup enabling human control of humanoids via mouse/keyboard or VR, with recording, replay, and multi-view rendering for evaluation and data collection.
• Robot and humanoid collaboration: study two tasks—Social Navigation (robot finds/follows a humanoid with safe distance) and Social Rearrangement (robot and humanoid jointly rearrange objects)—using learned and heuristic baselines.
• Baseline evaluation: compare end-to-end RL, heuristic planners, and population-based learning approaches across unseen scenes and collaborators to measure generalization and emergent cooperation.

Experimental results

Research questions

• RQ1 How can we model fast, diverse, and realistic humanoid avatars suitable for training embodied AI agents in home-like environments?
• RQ2 What is the impact of human-in-the-loop evaluation on estimating policy performance with real human partners?
• RQ3 How do learned and heuristic policies perform in social navigation and social rearrangement across unseen environments and unseen collaborators?
• RQ4 Can emergent cooperative behaviors (e.g., yielding space, task partitioning) improve efficiency and safety in human-robot collaboration?

Key findings

• End-to-end RL policies can learn cooperative behaviors like yielding space and adjusting motion to follow humanoids, achieving competitive success and collision metrics in Social Navigation.
• End-to-end RL and sensory ablations show varying importance of GPS, depth, and detector sensors, highlighting the role of different inputs before/after locating the humanoid.
• In Social Rearrangement, population-based training with multiple collaborators improves generalization to unseen partners, with Plan-Pop variants showing robust ZSC performance.
• Human-in-the-loop evaluations indicate that HITL-based automated assessments align with relative policy rankings when tested with real humans, aiding scalable benchmarking.
• Habitat 3.0 achieves high frame rates (e.g., 1191 FPS across 16 environments on a single GPU for robot-humanoid setups) while maintaining visual fidelity through caching and offline rig/load strategies.
• A modular two-layer policy (high-level skill selection plus low-level skills) supports efficient coordination between robot and humanoid across diverse scenes and partners.