[Paper Review] Habitat 2.0: Training Home Assistants to Rearrange their Habitat

Andrew Szot, Alexander Clegg+19 more|arXiv (Cornell University)|Jun 28, 2021

Reinforcement Learning in Robotics83 references25 citations

TL;DR

The paper introduces Habitat 2.0 (H2.0) with ReplicaCAD, a fast physics-enabled simulator and HAB benchmark to study long-horizon home-rearrangement tasks, comparing end-to-end RL policies with classical sense-plan-act pipelines and revealing hierarchical RL advantages and SPA brittleness.

ABSTRACT

We introduce Habitat 2.0 (H2.0), a simulation platform for training virtual robots in interactive 3D environments and complex physics-enabled scenarios. We make comprehensive contributions to all levels of the embodied AI stack - data, simulation, and benchmark tasks. Specifically, we present: (i) ReplicaCAD: an artist-authored, annotated, reconfigurable 3D dataset of apartments (matching real spaces) with articulated objects (e.g. cabinets and drawers that can open/close); (ii) H2.0: a high-performance physics-enabled 3D simulator with speeds exceeding 25,000 simulation steps per second (850x real-time) on an 8-GPU node, representing 100x speed-ups over prior work; and, (iii) Home Assistant Benchmark (HAB): a suite of common tasks for assistive robots (tidy the house, prepare groceries, set the table) that test a range of mobile manipulation capabilities. These large-scale engineering contributions allow us to systematically compare deep reinforcement learning (RL) at scale and classical sense-plan-act (SPA) pipelines in long-horizon structured tasks, with an emphasis on generalization to new objects, receptacles, and layouts. We find that (1) flat RL policies struggle on HAB compared to hierarchical ones; (2) a hierarchy with independent skills suffers from 'hand-off problems', and (3) SPA pipelines are more brittle than RL policies.

Motivation & Objective

• Create interactive, photo-realistic, home-scale environments with movable articulated objects for rearrangement tasks.
• Develop a high-performance physics-enabled simulator to enable large-scale RL and SPA experimentation.
• Provide a benchmark (HAB) to evaluate generalization to unseen objects, receptacles, and layouts.
• Systematically compare end-to-end reinforcement learning policies with classical sense-plan-act pipelines in long-horizon tasks.
• Analyze generalization, sensor reliance, and motion-planning integration to guide future embodied AI research.

Proposed method

• ReplicaCAD: an artist-authored, interactive 3D dataset of apartments with articulated objects (e.g., drawers, fridges) and 900+ hours of artist effort, designed to match real spaces and enable rearrangement experiments.
• Habitat 2.0: a high-performance physics-enabled simulator with localized physics, interleaved rendering/physics, and reuse of assets to achieve up to 26,000 SPS on 8 GPUs, enabling 850x real-time performance.
• Home Assistant Benchmark (HAB): a suite of tasks (tidy the house, prepare groceries, set the table) where a Fetch mobile manipulator rearranges objects from initial to target receptacles, with GeometricGoal-style specifications.
• Two experimental paradigms: monolithic end-to-end RL policies and classical sense-plan-act (SPA) pipelines, including a privileged SPA baseline with complete scene knowledge.
• Integration with OMPL for motion planning to enable fair comparisons between learned policies and classical planning approaches.

Experimental results

Research questions

• RQ1 How well do end-to-end RL policies scale to long-horizon home-rearrangement tasks?
• RQ2 Does a hierarchical RL approach outperform flat monolithic policies on HAB-like tasks?
• RQ3 How does SPA compare to RL in terms of robustness and generalization to unseen objects, receptacles, and layouts?
• RQ4 What generalization limits exist for both learning and planning approaches when faced with new objects, receptacles, and apartment configurations?
• RQ5 What are the sensor and planning requirements that influence performance and generalization in home-scale manipulation tasks?

Key findings

• Flat RL policies can learn diverse skills but struggle to chain them for long-horizon HAB tasks without a proper hierarchy.
• Hierarchical RL with a perfect task planner improves long-horizon performance but can suffer from hand-off problems between skills.
• SPA pipelines are brittle in complex, cluttered environments and can be outperformed by hierarchical learning approaches under certain generalization conditions.
• Monolithic RL generalizes to new layouts fairly well but shows notable drops with unseen objects and receptacles, highlighting object-level generalization challenges.
• SPA-Priv (privileged information) improves over SPA but still does not fully close the gap to learned hierarchical approaches in unseen-receptacle scenarios.